Kafka vs RabbitMQ: a log, or a queue?

Kafka is an append-only log. Messages are written to partitions and kept (for days, or forever); each consumer tracks its own offset — its position in the log. Many consumers read the same messages independently, and anyone can rewind and replay history. RabbitMQ is a message broker: a producer sends to an exchange, the broker routes it to queues by rules, a consumer takes it, acknowledges, and it’s deleted. The broker does the thinking; the message is transient.

The throughput gap isn't an implementation detail — it falls directly out of the two systems' core designs. RabbitMQ's broker does real work per message: it evaluates routing rules against an exchange, decides which queue(s) a message belongs in, tracks per-message acknowledgment state, and often persists to disk per-message for durability. That's real CPU and I/O work, multiplied by every single message, and it's exactly what makes RabbitMQ's routing so flexible.

Kafka sidesteps almost all of that. A producer writes to a partition, which is nothing more than an append to a sequential log file — the cheapest possible disk operation, and one modern disks (especially SSDs) handle at extraordinary rates. Kafka doesn't inspect message content to route it (the producer already picked the partition), doesn't track per-message state (consumers track their own offset, not the broker), and batches aggressively — grouping many messages into one disk write and one network round-trip. The result is a system built almost entirely around sequential I/O and batching, which is why a modest Kafka cluster can sustain millions of messages per second while a comparably-sized RabbitMQ cluster tops out dramatically lower. The lesson generalizes: RabbitMQ's intelligence lives in the broker per-message; Kafka pushes that intelligence to the edges (producers choose partitions, consumers choose what to read) and keeps the hot path as close to "just append to a file" as possible.

Say an e-commerce checkout needs to notify five downstream services when an order completes: inventory, billing, shipping, analytics, and fraud review. With RabbitMQ, you'd publish one message to a fanout (or topic) exchange, and RabbitMQ copies it into five separate queues — one per consumer. Each service processes and acknowledges independently; once all five have acked their copies, the message's job is done and it's gone. If a sixth service launches next quarter needing the same event, you bind a new queue to the exchange going forward — but it only sees events from the moment it started listening. There's no "replay the last 30 days of orders" without a separate mechanism.

With Kafka, the order-completed event is appended once to an orders topic partition. All five services subscribe independently and each tracks its own offset into that same, single copy of the log — Kafka doesn't duplicate the message per consumer the way RabbitMQ's fanout does. When fraud review ships a new ML model next quarter and needs to re-score the last 90 days of orders, it simply resets its consumer offset to 90 days ago and re-reads the same log everyone else is still consuming live from — no special replay infrastructure, because replay is just "read from an earlier offset," the same operation as normal consumption.