Learning series · 5 episodes · Senior / Staff / Principal
The stream keeps moving
Six lessons about producer buffering, partition expansion, hot consumers, poison records, retention gaps and broker fairness.
Start series →Follow the decisions in order
- Part 1 of 6
Senior
Publishing got faster. Why are urgent events arriving late?Choose producer batching policies by tracing buffering and end-to-end event freshness, including quiet-partition behavior.
- Part 2 of 6
Principal · AI Engineering reading
The newer customer update was processed first. Why did the old value win?Identify cross-partition key remapping during count changes and design cutover that preserves per-entity order.
- Part 3 of 6
Staff
We started 80 consumers. Why is one partition still two hours behind?Diagnose partition-level limits and per-key skew before claiming more consumer instances add throughput.
- Part 4 of 6
Staff
We quarantined the bad event. Which Kafka offset can we safely commit?Choose quarantine, repair and progress policy from per-key dependencies rather than applying a generic DLQ rule.
- Part 5 of 6
Staff
The consumer was down eight days. Where did day one go?Treat retention as a replay and recovery contract, distinguishing retained offsets from recoverable historical events.
- Part 6 of 6
Principal
One tenant published three times more. Why did every team slow down?Design end-to-end per-tenant quotas, admission and physical isolation while preserving other workloads under shared broker saturation.