The big idea
What is a message queue?
One service produces events much faster than another can handle them — or you want many services to react to the same event without the producer knowing they exist. Calling each consumer directly is brittle: a slow or dead consumer drags the producer down with it.
Put a durable log in the middle. Producers append; consumers read at their own speed. The producer fires and forgets; consumers come and go, replay history, and never block the producer. This one primitive underpins event-driven systems everywhere.
How to read this: We build it one piece at a time — a problem, then the smallest fix. Watch the diagram grow into a Kafka-style cluster. Hit Begin.
Step 1 · The core
An append-only log
A producer needs to hand off a message reliably, and a consumer needs to read messages in order, possibly long after they were written. A delete-on-read queue forgets too soon and can’t support replay or multiple readers.
Design decision: Consumers read in order long after a write, and multiple readers want the same stream. Model the topic as…
The call: An append-only commit log with monotonic offsets. — Writes only append (sequential disk I/O — shockingly fast), nothing is deleted on read, and order is implicit in the offset. Any number of consumers read it independently.
Model the topic as an append-only commit log: writes only ever append to the end and get a monotonically increasing offset. Nothing is deleted on read. Sequential disk appends are shockingly fast, and order is implicit in the offset.
The log is the database: An ordered, immutable, replayable sequence is a deceptively powerful abstraction. Because reads don’t mutate it, any number of consumers can read the same log independently.
Step 2 · Beyond one disk
Partitions for parallelism
A single log is capped by one machine’s disk and CPU — both for write throughput and for how fast consumers can read. One ordered log can’t scale horizontally.
Design decision: One ordered log is capped by a single machine’s disk and CPU. How do you scale throughput?
The call: Split the topic into partitions, routed by key hash. — Each partition is its own log on a possibly different broker; hashing a key keeps related events ordered together while unrelated events spread out. Throughput scales with partition count.
Split the topic into partitions, each its own log on a possibly different broker. A Partitioner routes each message — usually by hashing a key — so the same key always lands in the same partition. Throughput scales with partition count.
Order is per-partition: Kafka guarantees order within a partition, not across them. Choosing a good key (user id, order id) keeps related events ordered while letting unrelated events spread across partitions for parallelism.
Step 3 · Many readers, no deletes
Consumer groups & offsets
If the log never deletes, how does a consumer know what it has and hasn’t read? And how do ten instances of a service share the work without each processing every message?
Design decision: The log never deletes. How do consumers track progress and share work without each reading everything?
The call: Consumer groups split partitions; each group stores an offset cursor. — Partitions are divided among a group’s members (processed once per group), and a server-side offset records position — so a crash resumes exactly where it left off and a rewind replays. Different groups read the same log independently.
Consumers join a group; the group’s partitions are divided among its members so each message is processed once per group. Each group tracks its position with an offset — a cursor stored by the broker. Different groups read the same log at totally different positions.
The cursor, not the data, moves: Reading is just advancing an offset. That makes replay trivial (rewind the offset) and lets independent teams consume the same stream without stepping on each other.
Step 4 · Don’t lose data
Replication
Partitions live on disks, and disks (and whole brokers) die. If a partition exists on only one broker, its death means permanent data loss and downtime for everyone reading or writing it.
Design decision: A partition lives on a disk, and disks die. How do you not lose committed data?
The call: Replicate each partition: a leader + in-sync replicas, ack after replication. — The leader takes reads/writes; in-sync replicas mirror it, and a write commits only once enough replicas have it (acks=all). Losing the leader loses nothing — an ISR is promoted.
Replicate each partition across several brokers. One is the leader (takes all reads/writes); the others are in-sync replicas that mirror it. A write is “committed” once enough replicas have it, so losing the leader loses nothing.
Leader + ISR: The in-sync replica set is the safety margin. Acknowledge writes only after they’re replicated (acks=all) to trade a little latency for durability you can actually trust.
Step 5 · When a broker dies
Leader election
A leader broker crashes mid-flight. Someone has to notice, promote a healthy in-sync replica to leader, and tell every producer and consumer where to go now — without two brokers both thinking they’re the leader.
Design decision: The leader broker for a partition crashes mid-flight. How do clients keep working?
The call: A Controller elects a new leader from the ISR over a consensus store. — The controller watches health and promotes an in-sync replica via consensus (ZooKeeper/KRaft), then propagates metadata so clients transparently reconnect. Consensus is needed only for who-leads, not every message.
A Controller watches broker health and runs leader election over a consensus store (ZooKeeper, or Kafka’s own KRaft). It promotes an in-sync replica and propagates the new metadata, so clients transparently reconnect to the new leader.
Consensus for metadata: You don’t need consensus on every message — just on who leads each partition. Keeping that small, strongly-consistent decision separate from the high-volume data path is what makes the whole thing scale.
Step 6 · The log can’t grow forever
Retention & compaction
An append-only log that never forgets eventually fills every disk. But you also can’t blindly delete — some consumers replay history, and some topics need the latest value per key kept indefinitely.
Apply a retention policy: drop log segments older than N days or beyond a size cap. For keyed topics, use log compaction to keep only the most recent message per key, so the log becomes a compact snapshot of current state.
Two ways to forget: Time/size retention suits event streams (“keep 7 days”). Compaction suits changelogs (“keep the latest state per key forever, drop the history”). Pick per topic.
Step 7 · What can it promise?
Delivery guarantees
Networks drop acks and clients crash mid-batch, so producers retry and consumers reprocess — producing duplicates. Applications need to know what the system actually guarantees.
Design decision: Retries and crashes cause reprocessing. What guarantee do you offer, and how do you avoid duplicates?
The call: At-least-once by default; idempotent producers + transactions for exactly-once. — Retry until acked (accepting dupes), then dedup by producer-id + sequence and use transactions to commit write+offset atomically. Consumers commit offsets only after processing — that ordering defines the guarantee.
Default to at-least-once: retry until acknowledged, accept possible duplicates. Layer idempotent producers (dedup by producer id + sequence) and transactions (atomic write-plus-offset-commit) to reach exactly-once within Kafka. Consumers commit offsets only after successfully processing.
Commit after processing: The ordering of “do the work” and “commit the offset” decides your guarantee. Commit after work = at-least-once; the opposite = at-most-once. Exactly-once needs idempotency or transactions on top.
You did it
You just designed a message queue.
- An append-only commit log — ordered, immutable, replayable.
- Partitions split the log across brokers; a key-hash keeps order per partition.
- Consumer groups + server-side offsets: read once per group, rewind to replay.
- Replication with a leader + in-sync replicas so no committed write is lost.
- A controller runs leader election over a consensus store on broker failure.
- Retention and compaction keep the log from growing without bound.
- At-least-once by default; idempotent producers + transactions for exactly-once.