Kafka, and the
log that isn't a queue.

A traditional message queue delivers a message to one consumer and then deletes it — the message is consumed and gone. Kafka is different: it's a distributed append-only log. Producers append records to the end; the records are retained (for days, or forever); and each consumer reads forward at its own pace, tracking where it is rather than removing anything.

That one design choice unlocks Kafka's superpowers: records can be re-read (replay history, reprocess after a bug, bootstrap a new system), and many independent consumers can each read the whole stream — analytics, a search indexer, and a fraud detector all consuming the same events without interfering. The queue mindset ("deliver and delete") misses all of it.

Records go into topics (named streams, like orders or clicks). For scale, a topic is split into partitions — each an independent, ordered, append-only sequence. Every record in a partition gets a monotonically increasing offset: its position in that partition's log. A consumer's progress is just "I'm at offset 4,201 in partition 2."

Partitions are the heart of Kafka: they're the unit of parallelism, of ordering, and of distribution (partitions of a topic spread across the cluster's brokers). Almost every Kafka behavior traces back to "it's per-partition."

Producers append records to a topic (choosing a partition, usually by key — next section). Consumers read them. The clever part is the consumer group: a set of consumers that cooperatively read a topic, where Kafka assigns each partition to exactly one consumer in the group. Add consumers to a group and Kafka rebalances partitions across them — that's how you scale processing horizontally.

The catch: partitions cap your parallelism. A group can have at most as many active consumers as the topic has partitions; extra consumers sit idle. Meanwhile, different consumer groups are fully independent — each reads the entire stream with its own offsets — which is exactly what lets many applications consume the same topic without stepping on each other.

All three moving parts are arithmetic. Parallelism is a min, the key picks the partition by hash, and lag only drains when you consume faster than you produce:

Eight consumers on four partitions? Four work, four idle. Consume slower than you produce? Lag never drains. The runnable version below computes all three.

Here's the guarantee people most often get wrong: Kafka guarantees ordering only within a partition, never across a whole topic. Records in partition 2 are strictly ordered; records in partition 2 versus partition 0 have no defined order relative to each other, because they're processed in parallel.

So how do you keep related records in order — say, all events for one order id? You give them the same partition key. Kafka hashes the key to pick a partition, so all records with the same key land in the same partition and are therefore ordered. Same customer → same partition → in order. This is the central design decision when using Kafka: pick a key that groups the things that must stay ordered, while spreading load across partitions overall.

Each partition is replicated across several brokers for durability. One replica is the leader (handles all reads and writes for that partition); the others are followers that copy the leader's log. If the leader broker dies, a follower is promoted, so the partition survives machine failure.

The key concept is the in-sync replicas (ISR) — the set of replicas fully caught up with the leader. A producer can require that a write be acknowledged by all in-sync replicas (acks=all) before it's considered committed, guaranteeing the record survives the loss of any single broker. This is Kafka's durability–latency dial: acks=all is safest but waits for replicas; acks=1 is faster but can lose the latest writes if the leader dies before followers copy them.

How many times might a record be processed? Kafka supports three semantics, and the honest default is the middle one.

Most systems run at-least-once and make their consumers idempotent — safe to process the same record twice — because that's simpler and cheaper than true exactly-once. Kafka does offer exactly-once (via idempotent producers and transactions), but you should reach for idempotent at-least-once first and only pay for exactly-once when duplicates are genuinely unacceptable.

Because Kafka keeps records, it needs a rule for how long. Time/size retention keeps records for a window (e.g. 7 days) or up to a size, then deletes the oldest — perfect for event streams where old events stop mattering. Log compaction is different: it keeps at least the latest record per key, discarding superseded older values. That turns a topic into a durable, replayable snapshot of "the current value for every key" — ideal for changelog/state topics (a user's latest profile, an account's latest balance).

Compaction is why Kafka can back stateful systems: a compacted topic is a rebuildable source of truth for current state, not just a firehose of transient events.

Kafka isn't always the answer. Use a traditional queue (RabbitMQ, SQS) when you want simple task distribution — deliver each job to one worker, with per-message acknowledgement, priorities, and easy retall/dead-letter handling — and you don't need retention or replay. Queues are simpler for "work to be done."

Reach for Kafka when you have high-throughput event streams that multiple systems consume, when you need to replay history or reprocess, when ordering per key matters, or when the log itself is a source of truth (event sourcing, stream processing, data pipelines). The tell: many readers of the same durable, ordered stream → Kafka; distribute discrete tasks to workers → a queue.