The whole design, in writing
Learn system design by building a distributed message queue like Kafka step by step. An interactive guide covering the append-only log, partitions, consumer groups and offsets, replication, leader election, retention, and delivery guarantees.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a message queue?
One service produces events much faster than another can handle them — or you want many services to react to the same event without the producer knowing they exist. Calling each consumer directly is brittle: a slow or dead consumer drags the producer down with it.
Put a durable log in the middle. Producers append; consumers read at their own speed. The producer fires and forgets; consumers come and go, replay history, and never block the producer. This one primitive underpins event-driven systems everywhere.
What the new pieces do
- Producersproducer
- Services that emit a firehose of events — clicks, orders, metrics — and want to hand them off and move on, without waiting for anyone to read them.
Step 1 · The core
An append-only log
A producer needs to hand off a message reliably, and a consumer needs to read messages in order, possibly long after they were written. A delete-on-read queue forgets too soon and can’t support replay or multiple readers.
Consumers read in order long after a write, and multiple readers want the same stream. Model the topic as…
Delete-on-read forgets too soon: no replay, and a second consumer can’t read what the first consumed. You lose history and multi-reader support.
Random inserts/deletes mean index churn and locking, and you’d hand-roll ordering and cursors. A purpose-built log is far faster and simpler for an ordered stream.
Writes only append (sequential disk I/O — shockingly fast), nothing is deleted on read, and order is implicit in the offset. Any number of consumers read it independently.
Model the topic as an append-only commit log: writes only ever append to the end and get a monotonically increasing offset. Nothing is deleted on read. Sequential disk appends are shockingly fast, and order is implicit in the offset.
- append-onlywrite path
- sequentialdisk I/O
- offsetmessage id
What the new pieces do
- Brokerbackend
- The server that accepts writes from producers and serves reads to consumers. In a cluster, many brokers split the work; here we start with one.
- Commit Loglog
- The heart of it: an immutable, append-only sequence of messages on disk. Writes are just appends; each message gets a monotonically increasing offset.
Back of the envelope
- append = sequential disk write
- far faster than random I/O — near memory speeds
- offset = position in the log
- order is implicit, no extra index needed
- reads don’t mutate
- N consumers read one log independently
Step 2 · Beyond one disk
Partitions for parallelism
A single log is capped by one machine’s disk and CPU — both for write throughput and for how fast consumers can read. One ordered log can’t scale horizontally.
One ordered log is capped by a single machine’s disk and CPU. How do you scale throughput?
Vertical scaling hits a ceiling and is still one machine — one disk’s write rate, one CPU’s read rate. You need to spread a topic across machines.
Each partition is its own log on a possibly different broker; hashing a key keeps related events ordered together while unrelated events spread out. Throughput scales with partition count.
Concurrent writers to one shared log destroy the cheap sequential-append property and reintroduce coordination. Order and speed both come from each partition having a single writer.
Split the topic into partitions, each its own log on a possibly different broker. A Partitioner routes each message — usually by hashing a key — so the same key always lands in the same partition. Throughput scales with partition count.
- Npartitions = parallelism
- key-hashrouting
- per-partitionordering
What the new pieces do
- Partitionerservice
- Decides which partition an event lands in, usually by hashing a key. Same key → same partition → ordered together.
Back of the envelope
- throughput ∝ partition count
- add partitions to add parallelism
- same key ⇒ same partition
- related events stay ordered
- order is per-partition only
- not a global order across the topic
Step 3 · Many readers, no deletes
Consumer groups & offsets
If the log never deletes, how does a consumer know what it has and hasn’t read? And how do ten instances of a service share the work without each processing every message?
The log never deletes. How do consumers track progress and share work without each reading everything?
That’s back to a delete-on-read queue — no replay, and only one consumer can ever see a message. The log’s whole value is keeping data and letting readers track their own position.
Re-scanning the whole log per consumer is hugely wasteful with no shared progress. You need a stored cursor, not a full re-read.
Partitions are divided among a group’s members (processed once per group), and a server-side offset records position — so a crash resumes exactly where it left off and a rewind replays. Different groups read the same log independently.
Consumers join a group; the group’s partitions are divided among its members so each message is processed once per group. Each group tracks its position with an offset — a cursor stored by the broker. Different groups read the same log at totally different positions.
- per-groupcursor
- 1×delivery / group
- rewindto replay
What the new pieces do
- Consumer Groupconsumer
- Services that read the stream at their own pace — a fraud checker, a search indexer, a data warehouse — each independently, each possibly slow.
- Offsetsstore
- Where each consumer group has read up to. Stored server-side so a crashed consumer resumes exactly where it left off — the log itself is never mutated.
Step 4 · Don’t lose data
Replication
Partitions live on disks, and disks (and whole brokers) die. If a partition exists on only one broker, its death means permanent data loss and downtime for everyone reading or writing it.
A partition lives on a disk, and disks die. How do you not lose committed data?
Backups are minutes stale, so a broker death still loses recent writes and downs that partition. Durability must be synchronous with the write, not a periodic snapshot.
The leader takes reads/writes; in-sync replicas mirror it, and a write commits only once enough replicas have it (acks=all). Losing the leader loses nothing — an ISR is promoted.
Replicating every partition to every broker wastes enormous disk and write bandwidth. A few copies (typically 3) survive failures — you don’t need a copy on all of them.
Replicate each partition across several brokers. One is the leader (takes all reads/writes); the others are in-sync replicas that mirror it. A write is “committed” once enough replicas have it, so losing the leader loses nothing.
- ×3typical replication
- acks=alldurable write
- 0committed loss
What the new pieces do
- Replicas (ISR)store
- Each partition is copied to several brokers. One leader takes writes; in-sync followers mirror them, so a dead broker never loses committed data.
Back of the envelope
- ×3 replicas typical
- survives 2 broker failures
- acks=all ⇒ commit on ISR
- durability you can trust, slight latency cost
- 0 committed loss
- a promoted ISR already holds every committed write
Step 5 · When a broker dies
Leader election
A leader broker crashes mid-flight. Someone has to notice, promote a healthy in-sync replica to leader, and tell every producer and consumer where to go now — without two brokers both thinking they’re the leader.
The leader broker for a partition crashes mid-flight. How do clients keep working?
Clients have no consistent global view and could split-brain — two brokers each believing they lead. Leadership must be decided by one authority, not negotiated by clients.
The controller watches health and promotes an in-sync replica via consensus (ZooKeeper/KRaft), then propagates metadata so clients transparently reconnect. Consensus is needed only for who-leads, not every message.
Freezing the partition until a human fixes a machine destroys availability. The point of replication is to fail over in seconds, not wait for repair.
A Controller watches broker health and runs leader election over a consensus store (ZooKeeper, or Kafka’s own KRaft). It promotes an in-sync replica and propagates the new metadata, so clients transparently reconnect to the new leader.
What the new pieces do
- Controllerservice
- Tracks which brokers are alive and elects a new partition leader when one dies. Backed by a consensus store (ZooKeeper, or KRaft today).
Step 6 · The log can’t grow forever
Retention & compaction
An append-only log that never forgets eventually fills every disk. But you also can’t blindly delete — some consumers replay history, and some topics need the latest value per key kept indefinitely.
Apply a retention policy: drop log segments older than N days or beyond a size cap. For keyed topics, use log compaction to keep only the most recent message per key, so the log becomes a compact snapshot of current state.
What the new pieces do
- Retentionpolicy
- The log can’t grow forever. Old segments are dropped by age or size — or compacted to keep only the latest value per key.
Step 7 · What can it promise?
Delivery guarantees
Networks drop acks and clients crash mid-batch, so producers retry and consumers reprocess — producing duplicates. Applications need to know what the system actually guarantees.
Retries and crashes cause reprocessing. What guarantee do you offer, and how do you avoid duplicates?
Not retrying gives at-most-once — dropped messages on any failure. You can’t get reliability by giving up delivery; exactly-once is built on top of at-least-once, not instead of it.
Committing first means a crash after the commit but before the work silently loses the message (at-most-once). The safe default is to commit after processing.
Retry until acked (accepting dupes), then dedup by producer-id + sequence and use transactions to commit write+offset atomically. Consumers commit offsets only after processing — that ordering defines the guarantee.
Default to at-least-once: retry until acknowledged, accept possible duplicates. Layer idempotent producers (dedup by producer id + sequence) and transactions (atomic write-plus-offset-commit) to reach exactly-once within Kafka. Consumers commit offsets only after successfully processing.
You did it
You just designed a message queue.
Everything you assembled, in order
- An append-only commit log — ordered, immutable, replayable.
- Partitions split the log across brokers; a key-hash keeps order per partition.
- Consumer groups + server-side offsets: read once per group, rewind to replay.
- Replication with a leader + in-sync replicas so no committed write is lost.
- A controller runs leader election over a consensus store on broker failure.
- Retention and compaction keep the log from growing without bound.
- At-least-once by default; idempotent producers + transactions for exactly-once.