Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
Kafka Partition Calculator
A Kafka partition calculator. Enter your target throughput and the per-partition limits your producers and consumers can sustain, plus how many consumers you want to run in parallel, and get a recommended partition count with the reasoning — because partitions cap both throughput and consumer parallelism. Estimates topic storage from retention and replication too.
In Kafka, the partition count sets both the throughput ceiling of a topic and the most consumers that can read it in parallel. This calculator works out the partitions each constraint needs, takes the larger, and estimates how much data the topic will hold at your retention and replication.
What it computes
by_producer = ⌈target ÷ producer_per_partition⌉
by_consumer = ⌈target ÷ consumer_per_partition⌉
by_throughput = max(by_producer, by_consumer)
recommended = max(1, by_throughput, consumers)
storage_TB = target × retention_days × 86,400 × RF ÷ 1,000,000- target is the topic’s total throughput in MB/s at peak.
- producer_per_partition and consumer_per_partition are the MB/s one partition sustains on each side, measured on your hardware and settings. The consumer figure should include your processing time, not just fetch speed.
- consumers is how many consumers you want reading in parallel within one consumer group. Each partition goes to one consumer in a group, so consumers beyond the partition count sit idle.
- RF is the replication factor: every byte is stored on that many brokers.
The throughput part is the max(t/p, t/c) rule from Confluent’s well-known partition-sizing guidance, with the consumer count added as a third floor. The headline says which one won: “driven by consumer parallelism” when the consumer count exceeds the throughput need, “driven by throughput” otherwise. Storage uses decimal units, 1,000,000 MB to the TB, shown to one decimal.
A worked example
The defaults: a 100 MB/s topic, 10 MB/s per partition on both sides, 12 parallel consumers, 7 days of retention and a replication factor of 3.
- Producer side:
⌈100 ÷ 10⌉ = 10partitions. Consumer side: also 10. - By throughput:
max(10, 10) = 10. - Recommended:
max(1, 10, 12) = 12partitions, driven by consumer parallelism, because 12 consumers need 12 partitions for all of them to have work. - Storage:
100 × 7 × 86,400 × 3 = 181,440,000MB, shown as 181.4 TB.
| Change from the defaults | By throughput | Recommended | Driven by | Storage |
|---|---|---|---|---|
| None | 10 | 12 | consumer parallelism | 181.4 TB |
| Consumer rate falls to 4 MB/s | 25 | 25 | throughput | 181.4 TB |
| Target rises to 250 MB/s | 25 | 25 | throughput | 453.6 TB |
| Retention cut to 3 days | 10 | 12 | consumer parallelism | 77.8 TB |
| Replication factor 2 | 10 | 12 | consumer parallelism | 121.0 TB |
The second row is the common surprise. A consumer that does real work per message, such as a database write, often sustains far less per partition than the producer, and the slow side sets the count.
Assumptions and limits
- Per-partition rates are inputs, not facts. They depend on message size, batching, compression, acknowledgment settings, disks and network. Measure them with the perf-test tools that ship with Kafka or with your real consumer code.
- Even spread. Keyed messages land by a hash of the key, so a hot key concentrates load on one partition and one consumer, which no partition count fixes.
- No headroom. The recommendation is the bare minimum for today’s target. It adds nothing for growth or for catching up after consumer lag.
- Raw retained bytes. Storage treats the target MB/s as the bytes written to disk and ignores index files. Retention deletes only whole closed segments, so each partition keeps up to one extra segment, 1 GiB by default, beyond the window.
- No cluster limits. More partitions mean more open files, more replication traffic and slower recovery when a broker fails, and the tool does not check what your brokers can carry.
How to use the result
- Round up for growth. Partitions can be added later but never removed, and adding them remaps keys, which breaks per-key ordering across the change. Provision for the throughput you expect a year or two out.
- Cover peak consumer count. A consumer group scales out only to the partition count. If you plan to run 24 consumers at peak, you need at least 24 partitions even when throughput needs 10.
- Prefer a count that divides evenly. 12 partitions split evenly across 1, 2, 3, 4, 6 or 12 consumers, while 10 does not split evenly across 4. Even assignment stops one consumer carrying more than its share.
- Size disks from the storage figure. Divide by the broker count, then leave free space so a broker failure or a lagging consumer cannot fill the disks.
How it works
- Throughput need = max(⌈target ÷ producer rate⌉, ⌈target ÷ consumer rate⌉).
- Recommended partitions = max(1, throughput need, desired parallel consumers).
- Partition count caps consumer-group parallelism: a hard ceiling.
- Storage = throughput × retention × replication factor, in decimal TB.
Frequently asked questions
How many partitions should a Kafka topic have?
At least the larger of three numbers: target throughput divided by what one partition sustains on the producer side, the same on the consumer side, and the number of consumers you want reading in parallel. For a 100 MB/s topic at 10 MB/s per partition with 12 consumers, throughput needs 10 partitions and parallelism needs 12, so the answer is 12. Measure the per-partition rates on your own setup, then round up for growth, because partitions are hard to change later.
Why does partition count limit consumer parallelism?
Within a standard consumer group, Kafka assigns each partition to exactly one consumer, so a group can never have more active consumers than the topic has partitions. With 12 partitions, a 13th consumer in the same group gets no assignment and sits idle. Several consumer groups can each read the whole topic independently, but each is capped the same way. Newer Kafka releases add share groups, which let several consumers read one partition, but most deployments still size partitions for standard groups.
Can I add Kafka partitions later?
You can increase a topic’s partition count, but you can never decrease it. Adding partitions changes which partition each key hashes to, so new messages for a key can land on a different partition from its older messages, and per-key ordering breaks across the change. Existing data is not moved. For keyed topics it is much safer to provision enough partitions up front, with room for growth, than to add them under load once consumers fall behind.
How do I estimate Kafka storage for a topic?
Multiply throughput by the retention period in seconds and by the replication factor. A topic writing 100 MB/s for 7 days with a replication factor of 3 holds 100 × 604,800 × 3 MB, about 181.4 TB across the cluster. Replication multiplies the footprint because every partition is stored on that many brokers. Real disks need a little more, since retention deletes only whole closed segments, and less if your throughput figure is measured before producer compression.
Can a Kafka topic have too many partitions?
Yes. Each partition is a set of files on a broker and a unit of replication, so more partitions mean more open file handles, more replication traffic between brokers and more work when leadership moves after a broker fails. Producers also batch per partition, so spreading the same traffic over many partitions makes smaller batches and uses more memory. Consumer group rebalances take longer too. Size to your real throughput and parallelism plus growth, rather than picking a very large number just in case.
What per-partition throughput should I assume?
Measure it rather than assume it. The rate one partition sustains depends on message size, batching settings, compression, acknowledgment mode, disks and network, so published figures rarely transfer. Kafka ships kafka-producer-perf-test and kafka-consumer-perf-test for the raw rates. For the consumer side, test with your real processing code, because a consumer that writes each message to a database is usually far slower than the fetch itself, and the slower side sets the partition count.