Sharding Planner
A database sharding planner. Enter your total data size and query rate, and what a single shard can hold and serve, and get the number of shards you need — the larger of the storage and throughput requirements, with growth headroom — plus the per-shard load and a reminder of what resharding later costs.
Sharding is what you do when one machine can no longer hold your data or serve your traffic — and the shard count comes from whichever of those two walls you hit first. There are two independent requirements. Storage: total data divided by what a single shard can hold. Throughput: total QPS divided by what a single shard can serve. You need enough shards to clear both, so the count is the max of the two — a dataset that fits in three shards by size but demands eight by query load needs eight.
Then you add headroom, and it is not optional. Run shards at their limit and you have no slack for traffic spikes, for the uneven reality that some shards always run hotter than others, or for growth between now and your next capacity project. Planning to 60–70% of per-shard capacity buys that slack — which matters enormously because the alternative, adding shards later, is one of the most disruptive operations in distributed systems.
That is the part worth internalising: resharding is expensive. Change the shard count and the key-to-shard mapping changes, forcing a large fraction of your data to migrate across shards while you stay online. Consistent hashing softens the blow — only about 1/N of keys move when you add a node instead of nearly all of them — but it never makes it free. So you plan the count with headroom up front, and you pick a high-cardinality, evenly-accessed shard key, because even a perfect count cannot save you from a hot key hammering a single shard.
How it works
- Shards = max(storage need, throughput need), with headroom.
- Shows the per-shard data and QPS you are committing to.
- Reminds you resharding moves data — plan headroom up front.
- Consistent hashing limits, but never eliminates, data movement.
Frequently asked questions
How many shards do I actually need?
The larger of two requirements. Storage: total data divided by how much one shard can hold. Throughput: total QPS divided by how many queries one shard can serve. You need enough shards to satisfy both, so the answer is the max of the two, usually with growth headroom on top. A dataset that fits in three shards by size but needs eight by query load needs eight — the tighter constraint wins.
Why add headroom instead of sizing exactly?
Because running shards near 100% of their capacity leaves no room for traffic spikes, uneven key distribution (some shards run hotter than others), or growth before you can add capacity — and adding shards is disruptive. Planning to, say, 60–70% of per-shard capacity gives each shard breathing room and delays the next painful resharding. Sizing exactly to today’s load almost guarantees an emergency soon.
Why is resharding so costly?
When you change the number of shards, the mapping from key to shard changes, so a large fraction of your data has to physically move between shards while the system stays online — a slow, risky, load-heavy operation. Consistent hashing reduces how much data moves (only about 1/N of keys when adding one node), but it is never free. This is why you plan shards with headroom up front rather than resharding reactively under load.
How do I keep shards balanced?
Choose a shard key with high cardinality and even access, so data and load spread uniformly — a key like user_id usually beats something skewed like country. Watch for hot keys (a single celebrity account, a viral item) that can overwhelm one shard regardless of shard count; those need special handling like further splitting or caching. This planner sizes the count; a good shard key is what makes each shard actually carry its fair share.