The big idea

What is a message queue?

One service produces events much faster than another can handle them — or you want many services to react to the same event without the producer knowing they exist. Calling each consumer directly is brittle: a slow or dead consumer drags the producer down with it.

Put a durable log in the middle. Producers append; consumers read at their own speed. The producer fires and forgets; consumers come and go, replay history, and never block the producer. This one primitive underpins event-driven systems everywhere.

How to read this: We build it one piece at a time — a problem, then the smallest fix. Watch the diagram grow into a Kafka-style cluster. Hit Begin.

Step 1 · The core

An append-only log

A producer needs to hand off a message reliably, and a consumer needs to read messages in order, possibly long after they were written. A delete-on-read queue forgets too soon and can’t support replay or multiple readers.

Design decision: Consumers read in order long after a write, and multiple readers want the same stream. Model the topic as…

The call: An append-only commit log with monotonic offsets. — Writes only append (sequential disk I/O — shockingly fast), nothing is deleted on read, and order is implicit in the offset. Any number of consumers read it independently.

Model the topic as an append-only commit log: writes only ever append to the end and get a monotonically increasing offset. Nothing is deleted on read. Sequential disk appends are shockingly fast, and order is implicit in the offset.

The log is the database: An ordered, immutable, replayable sequence is a deceptively powerful abstraction. Because reads don’t mutate it, any number of consumers can read the same log independently.

Step 2 · Beyond one disk

Partitions for parallelism

A single log is capped by one machine’s disk and CPU — both for write throughput and for how fast consumers can read. One ordered log can’t scale horizontally.

Design decision: One ordered log is capped by a single machine’s disk and CPU. How do you scale throughput?

The call: Split the topic into partitions, routed by key hash. — Each partition is its own log on a possibly different broker; hashing a key keeps related events ordered together while unrelated events spread out. Throughput scales with partition count.

Split the topic into partitions, each its own log on a possibly different broker. A Partitioner routes each message — usually by hashing a key — so the same key always lands in the same partition. Throughput scales with partition count.

Order is per-partition: Kafka guarantees order within a partition, not across them. Choosing a good key (user id, order id) keeps related events ordered while letting unrelated events spread across partitions for parallelism.

Step 3 · Many readers, no deletes

Consumer groups & offsets

If the log never deletes, how does a consumer know what it has and hasn’t read? And how do ten instances of a service share the work without each processing every message?

Design decision: The log never deletes. How do consumers track progress and share work without each reading everything?

The call: Consumer groups split partitions; each group stores an offset cursor. — Partitions are divided among a group’s members (processed once per group), and a server-side offset records position — so a crash resumes exactly where it left off and a rewind replays. Different groups read the same log independently.

Consumers join a group; the group’s partitions are divided among its members so each message is processed once per group. Each group tracks its position with an offset — a cursor stored by the broker. Different groups read the same log at totally different positions.

The cursor, not the data, moves: Reading is just advancing an offset. That makes replay trivial (rewind the offset) and lets independent teams consume the same stream without stepping on each other.

Step 4 · Don’t lose data

Replication

Partitions live on disks, and disks (and whole brokers) die. If a partition exists on only one broker, its death means permanent data loss and downtime for everyone reading or writing it.

Design decision: A partition lives on a disk, and disks die. How do you not lose committed data?

The call: Replicate each partition: a leader + in-sync replicas, ack after replication. — The leader takes reads/writes; in-sync replicas mirror it, and a write commits only once enough replicas have it (acks=all). Losing the leader loses nothing — an ISR is promoted.

Replicate each partition across several brokers. One is the leader (takes all reads/writes); the others are in-sync replicas that mirror it. A write is “committed” once enough replicas have it, so losing the leader loses nothing.

Leader + ISR: The in-sync replica set is the safety margin. Acknowledge writes only after they’re replicated (acks=all) to trade a little latency for durability you can actually trust.

Step 5 · When a broker dies

Leader election

A leader broker crashes mid-flight. Someone has to notice, promote a healthy in-sync replica to leader, and tell every producer and consumer where to go now — without two brokers both thinking they’re the leader.

Design decision: The leader broker for a partition crashes mid-flight. How do clients keep working?

The call: A Controller elects a new leader from the ISR over a consensus store. — The controller watches health and promotes an in-sync replica via consensus (ZooKeeper/KRaft), then propagates metadata so clients transparently reconnect. Consensus is needed only for who-leads, not every message.

A Controller watches broker health and runs leader election over a consensus store (ZooKeeper, or Kafka’s own KRaft). It promotes an in-sync replica and propagates the new metadata, so clients transparently reconnect to the new leader.

Consensus for metadata: You don’t need consensus on every message — just on who leads each partition. Keeping that small, strongly-consistent decision separate from the high-volume data path is what makes the whole thing scale.

Step 6 · The log can’t grow forever

Retention & compaction

An append-only log that never forgets eventually fills every disk. But you also can’t blindly delete — some consumers replay history, and some topics need the latest value per key kept indefinitely.

Apply a retention policy: drop log segments older than N days or beyond a size cap. For keyed topics, use log compaction to keep only the most recent message per key, so the log becomes a compact snapshot of current state.

Two ways to forget: Time/size retention suits event streams (“keep 7 days”). Compaction suits changelogs (“keep the latest state per key forever, drop the history”). Pick per topic.

Step 7 · What can it promise?

Delivery guarantees

Networks drop acks and clients crash mid-batch, so producers retry and consumers reprocess — producing duplicates. Applications need to know what the system actually guarantees.

Design decision: Retries and crashes cause reprocessing. What guarantee do you offer, and how do you avoid duplicates?

The call: At-least-once by default; idempotent producers + transactions for exactly-once. — Retry until acked (accepting dupes), then dedup by producer-id + sequence and use transactions to commit write+offset atomically. Consumers commit offsets only after processing — that ordering defines the guarantee.

Default to at-least-once: retry until acknowledged, accept possible duplicates. Layer idempotent producers (dedup by producer id + sequence) and transactions (atomic write-plus-offset-commit) to reach exactly-once within Kafka. Consumers commit offsets only after successfully processing.

Commit after processing: The ordering of “do the work” and “commit the offset” decides your guarantee. Commit after work = at-least-once; the opposite = at-most-once. Exactly-once needs idempotency or transactions on top.

You did it

You just designed a message queue.

• A — n
• P — a
• C — o
• R — e
• A —
• R — e
• A — t