Learning series · 6 episodes · Staff / Principal
Designing Reliable Distributed Messaging Systems
Six connected engineering decisions: lease a job, not a business effect, partitioned append-only history, what a producer ACK proves, rebuild state from a keyed log, keep consumer ownership stable, translate progress across regions.
Start series →Follow the decisions in order
- Part 1 of 6
Staff
How would you design a distributed work queue like Amazon SQS with at-least-once delivery?Sharded storage, leases and dedupe windows.
- Part 2 of 6
Staff
How would you design a partitioned, replicated commit log like Apache Kafka?Segmented logs, ISR replication and offset reads.
- Part 3 of 6
Staff
How would you configure Kafka to prevent acknowledged messages from being lost during broker failures?acks, min.insync.replicas, replication factor and unclean elections.
- Part 4 of 6
Staff
How does Kafka log compaction work, and how would you use it to rebuild application state?Latest value per key, tombstones and changelogs.
- Part 5 of 6
Staff
How would you diagnose and fix repeated Kafka consumer-group rebalances that stop processing?Poll intervals, cooperative rebalancing and static membership.
- Part 6 of 6
Principal
How would you recover Kafka consumers after failing over to a mirrored cluster in another region?Offset translation and consumer position in mirrored clusters.