The whole design, in writing
Learn system design by building a distributed unique ID generator like Twitter Snowflake step by step. An interactive guide covering why auto-increment and UUIDs fall short, the timestamp/machine/sequence bit layout, time-sortable 64-bit IDs, machine-id assignment, clock skew, and alternatives.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why is this hard?
“Give me a unique ID” sounds trivial — until it must work across hundreds of machines, millions of times a second, with zero collisions, and ideally produce IDs that sort by time. A single counter can’t keep up, and randomness has its own costs.
Generate IDs locally on each machine with no per-ID coordination, by carving a 64-bit integer into fields — time, machine, sequence — that together guarantee uniqueness. This is Twitter’s Snowflake, and it’s mostly a packing problem.
What the new pieces do
- Serviceclient
- Any service creating a row, message or event that needs a unique identifier — millions of times a second, across many machines.
Step 1 · The naive options
Local, no round-trip
A database auto-increment is one counter — a single point of failure and a write bottleneck the whole fleet must queue behind. Calling a central service per ID adds a network hop to every insert.
You need unique IDs millions/sec across hundreds of machines. Where do you generate them?
One counter is a single write bottleneck and SPOF the whole fleet queues behind. It can’t keep up with millions/sec across many machines.
Correct and ordered, but it adds a network round-trip to every single insert and is itself a bottleneck/SPOF. The hot path can’t afford a hop per ID.
Each service generates IDs locally with no network call — speed is the point. The rest is just deciding how to fill 64 bits so they never collide.
Make the generator local: a library inside each service that produces a 64-bit ID in-process, instantly, with no network call. The rest of the design is just deciding how to fill those 64 bits so they never collide.
What the new pieces do
- ID Generatorbackend
- Runs as a library inside each service (or a nearby sidecar). It produces IDs locally with no network round-trip per ID — speed is the whole point.
- 64-bit IDid
- The three fields packed into one 64-bit integer: sign bit, timestamp, machine id, sequence. Compact, index-friendly, and time-sortable.
Back of the envelope
- 64-bit int vs 128-bit UUID
- half the size ⇒ smaller, faster DB indexes
- local mint = 0 network hops
- no per-ID round-trip on the hot path
- 1 counter ⇒ bottleneck + SPOF
- why DB auto-increment can’t scale the fleet
Step 2 · Put time first
Timestamp in the high bits
Random IDs (like a UUIDv4) are unique but unordered — inserting them scatters writes all over a database index, hurting cache locality and range queries. You lose all sense of “newer” vs “older”.
You want IDs that don’t scatter DB index writes and that sort by creation time. How?
Unique, but unordered — random inserts scatter across the index, hurting cache locality and range scans, and you lose newer-vs-older. Also twice the size of a 64-bit int.
Later IDs are numerically larger, so they’re roughly time-sortable and new inserts append to the end of the index. 41 bits of ms covers ~69 years from a custom epoch.
Time in the low bits means the most-significant bits don’t increase with time, so IDs don’t sort by creation order — defeating the whole reason to include time. Time must lead.
Place a millisecond timestamp in the top bits. Now IDs minted later are numerically larger, so they’re roughly time-sortable and new inserts append to the end of the index. 41 bits of milliseconds covers ~69 years from a custom epoch.
- 41 bitstimestamp
- ~69 yrsof ms
- k-sortedby time
What the new pieces do
- Timestampfield
- Milliseconds since a custom epoch in the high bits. Putting time first makes IDs increase over time, so they roughly sort by creation order.
Back of the envelope
- 41 bits of ms ≈ 69 years
- from a custom epoch, in the high bits
- time leads ⇒ k-sorted IDs
- new inserts append to the index tail
- ~k-sorted, not perfectly ordered
- clocks differ slightly across machines
Step 3 · Separate the machines
A machine ID field
Two machines minting an ID in the same millisecond would produce the same timestamp bits — and collide. Coordinating between them on every ID would re-introduce the network hop we just removed.
Two machines minting in the same millisecond share timestamp bits and collide. Fix without a network hop?
Per-ID coordination re-introduces the network round-trip you removed in step 1. Uniqueness should come from structure, not a conversation per ID.
Hope isn’t a guarantee — birthday-paradox collisions appear at high rates, and you’d need many random bits, bloating the ID. You want collision-free by construction.
Same millisecond, different machine bits → different IDs. Each node owns its slice of every millisecond, so they physically can’t collide — coordination happens once at assignment, not per ID. 10 bits = 1,024 generators.
Give each generator a unique machine ID in the middle bits. Same millisecond, different machines, different IDs — uniqueness by construction, with no per-ID coordination. 10 bits allows 1,024 distinct generators.
- 10 bitsmachine id
- 1,024generators
- 0per-ID coordination
What the new pieces do
- Machine IDfield
- A unique number per generator instance. Because each machine owns its slice, two machines in the same millisecond never collide — no coordination per ID.
Step 4 · Many IDs per millisecond
The sequence counter
One machine often needs more than one ID per millisecond. Timestamp + machine bits alone would repeat for every ID minted within the same millisecond on the same node.
One machine often needs many IDs in the same millisecond. Timestamp + machine bits would repeat. Fix?
Capping a node at one ID per millisecond (1,000/sec) is far too slow. You need many IDs within a millisecond without waiting on the clock each time.
A counter that increments per ID and resets each millisecond lets one machine mint up to 4,096 IDs/ms (12 bits, ~4M/sec). Only if it overflows do you wait for the next tick.
Thread ids aren’t bounded or unique across processes and can repeat, so they don’t guarantee uniqueness within a millisecond. A reset-per-ms counter is exact and compact.
Add a sequence counter in the low bits that increments per ID and resets each millisecond. With 12 bits, a single machine can mint 4,096 IDs per millisecond (~4M/sec). If it overflows, wait for the next millisecond tick.
- 12 bitssequence
- 4,096IDs / ms / node
- ~4Mper second
What the new pieces do
- Sequencefield
- A counter reset each millisecond, so one machine can mint up to 4,096 IDs in the same millisecond before it must wait for the clock to tick.
Back of the envelope
- 12 bits ⇒ 4,096 IDs / ms / node
- ~4M IDs/sec per machine
- [1 | 41 time | 10 node | 12 seq]
- the full 64-bit layout
- seq overflow ⇒ wait 1 ms tick
- the only stall on a single node
Step 5 · Hand out machine IDs
Assign node bits safely
The whole scheme breaks if two generators are accidentally given the same machine ID — then they collide every millisecond. So who assigns those 1,024 ids, and how do you avoid duplicates as machines come and go?
The scheme breaks if two generators get the same machine id. Who assigns the 1,024 ids?
Random picks collide (birthday paradox at 1,024 slots), and two colliding generators then clash every millisecond. Assignment must guarantee uniqueness, not gamble on it.
Each generator claims a free id once on startup and holds it via a lease. That’s the only coordination in the whole system — after it, every node runs independently and collision-free.
Static config works until autoscaling, re-IPing or cloning silently duplicates an id. A lease-based registry handles machines coming and going without accidental reuse.
Use an ID Registry: on startup each generator claims a free machine id from a coordination service (ZooKeeper/etcd) or static config, holding it via a lease. The only coordination in the entire system happens here, once per process.
What the new pieces do
- ID Registryservice
- Hands each generator a unique machine id on startup (via ZooKeeper/etcd or config), so no two instances ever share the same node bits.
Step 6 · When the clock lies
Clock skew & rewind
IDs trust the wall clock — but NTP can step the clock backward. If time moves back, a machine could reuse timestamps it already used and mint duplicate IDs. The one assumption Snowflake makes is the one thing clocks violate.
NTP can step the wall clock backward. What stops the generator minting duplicate IDs?
Rare isn’t never: one NTP correction can reuse a whole millisecond range of IDs, producing duplicates — the one failure this system must never have.
Time zones aren’t the issue — NTP corrections step the underlying wall clock backward regardless of zone. You must detect and guard rewind, not relabel the clock.
A clock guard remembers the last value used and stalls/refuses until the clock passes it — never issuing an ID for a time it already served. A tiny rare pause buys an ironclad no-duplicates guarantee.
A Clock Guard tracks the last timestamp used. If the clock goes backward, the generator refuses to mint (or briefly stalls) until the clock catches up past the last value — never issuing an ID for a time it has already served.
What the new pieces do
- Clock Guardguard
- Watches for the wall clock jumping backward (NTP correction). If time moves back, it refuses to mint until the clock catches up, preventing duplicate IDs.
Step 7 · Other ways
Alternatives & trade-offs
Snowflake gives unique, k-sorted IDs without coordination — but it leaks creation time (a privacy/competitive signal) and isn’t strictly monotonic across machines. Sometimes you need different properties.
Pick the tool for the job: a DB ticket server or range allocator (hand out blocks of ids per machine) gives strict ordering at the cost of a central component; UUIDv7 offers time-ordered randomness with no machine-id assignment. Snowflake wins when you want local, fast, sortable, and 64-bit.
What the new pieces do
- Ticket Serverstore
- An alternative: a database (or range allocator) that hands out monotonically increasing ids. Simpler, strongly ordered, but a potential bottleneck and SPOF.
You did it
You just designed an ID generator.
Everything you assembled, in order
- IDs are generated locally, in-process, with no per-ID network round-trip.
- A timestamp in the high bits makes 64-bit IDs roughly time-sortable.
- A machine-id field keeps same-millisecond IDs on different nodes disjoint.
- A per-millisecond sequence counter handles bursts (4,096 IDs/ms/node).
- A registry assigns unique machine ids once at startup — the only coordination.
- A clock guard refuses to mint when the wall clock rewinds, avoiding duplicates.
- Ticket servers, range allocators and UUIDv7 are alternatives with other trade-offs.