Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
Sharding Planner
A database sharding planner. Enter your total data size and query rate, and what a single shard can hold and serve, and get the number of shards you need — the larger of the storage and throughput requirements, with growth headroom — plus the per-shard load and a reminder of what resharding later costs.
Sharding is what you do when one machine can no longer hold your data or serve your traffic, and the shard count comes from whichever of those two walls you hit first. This planner sizes both walls with headroom, takes the larger, and shows what each shard and the whole cluster then carry.
What it computes
utilization = 1 − headroom / 100 at least 0.05
by storage = ⌈ total data / (per-shard GB × utilization) ⌉
by throughput = ⌈ total QPS / (per-shard QPS × utilization) ⌉
shards = max(1, by storage, by throughput)
per shard = total data / shards and total QPS / shards
nodes = shards × replication factor- Per-shard capacity is the data one shard can hold and the queries per second it can serve at its limit, measured or taken from benchmarks of your own setup.
- Headroom is the share of each limit you refuse to use, kept for spikes, skew and growth.
- Bound by names the requirement that set the count. A tie is reported as storage.
- Replication factor copies every shard, multiplying machines but not the shard count.
A worked example
The defaults are 4,000 GB of data and 120,000 QPS, with shards that hold 500 GB and serve 20,000 QPS, 30% headroom and 3 replicas.
- Utilization:
1 − 0.30 = 0.7, so each shard is planned to 350 GB and 14,000 QPS. - By storage:
4,000 / 350 = 11.4, rounded up to 12. - By throughput:
120,000 / 14,000 = 8.6, rounded up to 9. - Shards:
max(12, 9) = 12, bound by storage. - Per shard:
4,000 / 12 = 333 GBand120,000 / 12 = 10.0k QPS, which is 67% of the disk limit and 50% of the query limit. - With 3 replicas:
12 × 3 = 36 nodes.
| Change | By storage | By throughput | Shards | Bound by |
|---|---|---|---|---|
| Defaults | 12 | 9 | 12 | storage |
| No headroom | 8 | 6 | 8 | storage |
| QPS doubles to 240,000 | 12 | 18 | 18 | throughput |
| Shards hold 1,000 GB | 6 | 9 | 9 | throughput |
The last two rows show why both walls matter: fix the binding one and the other takes over.
Assumptions and limits
- Perfectly even spread. Every shard gets the same data and load. Real key distributions are skewed, and one hot key can saturate a shard on its own.
- Primaries do all the work. Read replicas that serve traffic are not credited, and write and read costs are not separated.
- Today’s numbers. There is no growth rate; plan with the data and traffic you expect at the end of the period, not now.
- Capacity is yours to supply. Per-shard limits depend on hardware, schema, indexes and query mix, so benchmark them.
How to use the result
- Pick a count you can grow from. Many systems create more logical partitions than machines, for example several per node, so adding hardware moves whole partitions instead of rehashing keys.
- Watch the binding limit. It is the metric to alert on and the one that triggers the next expansion.
- Choose the shard key before the count. High cardinality and even access matter more than any number this tool produces.
- Leave the migration budget. Resharding adds load while data moves, so start expanding while headroom remains.
How it works
- Shards = max(storage need, throughput need), each sized with headroom.
- Shows which limit binds and the per-shard data and QPS you commit to.
- Nodes = shards × replication factor.
- Consistent hashing limits, but never eliminates, data movement on resharding.
Frequently asked questions
How many shards do I actually need?
The larger of two requirements. Storage: total data divided by how much one shard can hold. Throughput: total queries per second divided by how many one shard can serve. You need enough shards to satisfy both, so the answer is the maximum of the two, after leaving headroom on each. With this tool’s defaults, 4,000 GB needs 12 shards by storage and 120,000 QPS needs 9 by throughput, so the plan is 12, bound by storage.
Why add headroom instead of sizing exactly?
Because shards running near full capacity have no room for traffic spikes, uneven key distribution, where some shards always run hotter than others, or growth before new capacity arrives, and adding shards is disruptive. Planning each shard to about 60 to 70% of its limit gives breathing room and delays the next resharding. The tool divides each limit by one minus the headroom, so the default 30% means planning to 70% of capacity.
Why is resharding so costly?
When the number of shards changes, the mapping from keys to shards changes, so a large fraction of the data has to move between machines while the system stays online. That is slow, adds load at the worst moment and is easy to get wrong. Consistent hashing or a fixed set of many small partitions reduces how much moves, to roughly one new shard’s share of the keys, but never to zero. Plan the count with headroom instead of resharding under pressure.
How do I keep shards balanced?
Choose a shard key with high cardinality and even access, so data and load spread uniformly; a key such as user ID usually beats something skewed such as country. Watch for hot keys, such as a celebrity account or a viral item, that can overwhelm one shard regardless of how many there are; those need special handling such as splitting the key or caching it. This planner sizes the count, and the shard key decides whether each shard really carries its share.
Do replicas count toward shard capacity?
Not in this calculation. The shard count is sized as if each shard’s primary holds its data and serves its share of the queries, and replication then multiplies the number of machines: 12 shards with three copies each is 36 nodes. If your replicas serve reads, the real throughput per shard is higher and you may need fewer shards for a read-heavy workload. Writes still go to one primary per shard, so size write-heavy systems as the tool does.