The whole design, in writing
Learn system design by building a distributed unique ID generator like Twitter Snowflake step by step. An interactive guide covering why auto-increment and UUIDs fall short, the timestamp/machine/sequence bit layout, time-sortable 64-bit IDs, machine-id assignment, clock skew, and alternatives.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why is this hard?
“Give me a unique ID” sounds trivial — until it must work across hundreds of machines, millions of times a second, with zero collisions, and ideally produce IDs that sort by time. A single counter can’t keep up, and randomness has its own costs.
Generate IDs locally on each machine with no per-ID coordination, by carving a 64-bit integer into fields — time, machine, sequence — that together guarantee uniqueness. This is Twitter’s Snowflake, and it’s mostly a packing problem.
What the new pieces do
- Serviceclient
- Any service creating a row, message or event that needs a unique identifier — millions of times a second, across many machines.
Step 1 · The naive options
Local, no round-trip
A database auto-increment is one counter — a single point of failure and a write bottleneck the whole fleet must queue behind. Calling a central service per ID adds a network hop to every insert.
You need unique IDs millions/sec across hundreds of machines. Where do you generate them?
One counter is a single write bottleneck and SPOF the whole fleet queues behind. It can’t keep up with millions/sec across many machines.
Correct and ordered, but it adds a network round-trip to every single insert and is itself a bottleneck/SPOF. The hot path can’t afford a hop per ID.
Each service generates IDs locally with no network call — speed is the point. The rest is just deciding how to fill 64 bits so they never collide.
Make the generator local: a library inside each service that produces a 64-bit ID in-process, instantly, with no network call. The rest of the design is just deciding how to fill those 64 bits so they never collide.
Why this piece earns its place
Worth naming what going in-process actually commits you to: the id format now ships on every service’s release cycle. The bit layout, the epoch and the clock guard are library code linked into each application, so the day any of them has to change you are running a fleet-wide redeploy, and for a long window old and new binaries mint ids side by side — which is only survivable if the change is compatible with ids already written. It also means one implementation per language your fleet speaks, and they have to agree bit for bit. If that is not realistic, the honest version of this design is a sidecar on each host: you keep the coordination-free property and a single implementation, and you pay a loopback call instead of a network round-trip. I would still start in-process, but I would keep the packing behind a small interface so moving to a sidecar is a swap rather than a rewrite.
What the new pieces do
- ID Generatorbackend
- Runs as a library inside each service (or a nearby sidecar). It produces IDs locally with no network round-trip per ID — speed is the whole point.
- 64-bit IDid
- The three fields packed into one 64-bit integer: sign bit, timestamp, machine id, sequence. Compact, index-friendly, and time-sortable.
Back of the envelope
- 64-bit int vs 128-bit UUID
- half the size ⇒ smaller, faster DB indexes
- local mint = 0 network hops
- no per-ID round-trip on the hot path
- 1 counter ⇒ bottleneck + SPOF
- why DB auto-increment can’t scale the fleet
Step 2 · Put time first
Timestamp in the high bits
Random IDs (like a UUIDv4) are unique but unordered — inserting them scatters writes all over a database index, hurting cache locality and range queries. You lose all sense of “newer” vs “older”.
You want IDs that don’t scatter DB index writes and that sort by creation time. How?
Unique, but unordered — random inserts scatter across the index, hurting cache locality and range scans, and you lose newer-vs-older. Also twice the size of a 64-bit int.
Later IDs are numerically larger, so they’re roughly time-sortable and new inserts append to the end of the index. 41 bits of ms covers ~69 years from a custom epoch.
Time in the low bits means the most-significant bits don’t increase with time, so IDs don’t sort by creation order — defeating the whole reason to include time. Time must lead.
Place a millisecond timestamp in the top bits. Now IDs minted later are numerically larger, so they’re roughly time-sortable and new inserts append to the end of the index. 41 bits of milliseconds covers ~69 years from a custom epoch.
Why this piece earns its place
The knob this step introduces is the epoch, and you set it exactly once. 41 bits of milliseconds is a window of about 69 years that starts counting from whatever zero point you pick, so the useful move is to set the epoch to the day the system first deploys rather than to the Unix epoch — every year that passed before your first id existed is a year burned off the far end of the format. Set it in the future instead and early elapsed times go negative, inverting the ordering you just bought. The part people miss is that the epoch stops being configuration the moment an id is persisted: it is baked into every id already written, so moving it later does not shift new ids into fresh space, it re-points them at ranges you have already issued. Record the epoch beside the bit layout, treat both as a frozen wire format, and put the expiry date in a calendar instead of discovering it.
- 41 bitstimestamp
- ~69 yrsof ms
- k-sortedby time
What the new pieces do
- Timestampfield
- Milliseconds since a custom epoch in the high bits. Putting time first makes IDs increase over time, so they roughly sort by creation order.
Back of the envelope
- 41 bits of ms ≈ 69 years
- from a custom epoch, in the high bits
- time leads ⇒ k-sorted IDs
- new inserts append to the index tail
- ~k-sorted, not perfectly ordered
- clocks differ slightly across machines
Step 3 · Separate the machines
A machine ID field
Two machines minting an ID in the same millisecond would produce the same timestamp bits — and collide. Coordinating between them on every ID would re-introduce the network hop we just removed.
Two machines minting in the same millisecond share timestamp bits and collide. Fix without a network hop?
Per-ID coordination re-introduces the network round-trip you removed in step 1. Uniqueness should come from structure, not a conversation per ID.
Hope isn’t a guarantee — birthday-paradox collisions appear at high rates, and you’d need many random bits, bloating the ID. You want collision-free by construction.
Same millisecond, different machine bits → different IDs. Each node owns its slice of every millisecond, so they physically can’t collide — coordination happens once at assignment, not per ID. 10 bits = 1,024 generators.
Give each generator a unique machine ID in the middle bits. Same millisecond, different machines, different IDs — uniqueness by construction, with no per-ID coordination. 10 bits allows 1,024 distinct generators.
Why this piece earns its place
Ten bits for the machine field is a choice, not a law, and the thing to size it against is concurrently live generator processes, not hosts. Containers put several processes on one machine, and a rolling deploy runs the old and the new version at the same time, so peak demand for slots sits well above steady state — size for that overlap or you exhaust the 1,024 slots during a deploy rather than during growth. Widening the field is paid for out of the other two, and both bills are visible: bits taken from the sequence cut how many ids a node can mint inside a millisecond, bits taken from the timestamp cut how many years the format lasts. What makes the split worth settling carefully is that it is fleet-wide and permanent. Nothing in the id says where the boundaries are, so a service that decodes with a different split reads a different number out of the same bytes, and nothing complains.
- 10 bitsmachine id
- 1,024generators
- 0per-ID coordination
What the new pieces do
- Machine IDfield
- A unique number per generator instance. Because each machine owns its slice, two machines in the same millisecond never collide — no coordination per ID.
Step 4 · Many IDs per millisecond
The sequence counter
One machine often needs more than one ID per millisecond. Timestamp + machine bits alone would repeat for every ID minted within the same millisecond on the same node.
One machine often needs many IDs in the same millisecond. Timestamp + machine bits would repeat. Fix?
Capping a node at one ID per millisecond (1,000/sec) is far too slow. You need many IDs within a millisecond without waiting on the clock each time.
A counter that increments per ID and resets each millisecond lets one machine mint up to 4,096 IDs/ms (12 bits, ~4M/sec). Only if it overflows do you wait for the next tick.
Thread ids aren’t bounded or unique across processes and can repeat, so they don’t guarantee uniqueness within a millisecond. A reset-per-ms counter is exact and compact.
Add a sequence counter in the low bits that increments per ID and resets each millisecond. With 12 bits, a single machine can mint 4,096 IDs per millisecond (~4M/sec). If it overflows, wait for the next millisecond tick.
Why this piece earns its place
This step quietly requires that reading the clock, comparing it against the last timestamp, bumping the counter and packing the result happen as one indivisible operation. They are separate pieces of state, and a node minting from many threads will interleave them — two threads read the same millisecond, both read the same counter, both emit the same id. So the whole mint sits under one lock or one compare-and-swap, which also makes the generator a contention point inside the process at precisely the moment it is busiest. The overflow path deserves the same care: past 4,096 ids in a millisecond the node has nothing to do but wait for the clock to tick, and a naive spin there burns a core and turns a capacity ceiling into a latency cliff for everything else on the box. If one process genuinely needs more than that, do not widen the sequence — give it a second machine id and mint from two independent generators.
- 12 bitssequence
- 4,096IDs / ms / node
- ~4Mper second
What the new pieces do
- Sequencefield
- A counter reset each millisecond, so one machine can mint up to 4,096 IDs in the same millisecond before it must wait for the clock to tick.
Back of the envelope
- 12 bits ⇒ 4,096 IDs / ms / node
- ~4M IDs/sec per machine
- [1 | 41 time | 10 node | 12 seq]
- the full 64-bit layout
- seq overflow ⇒ wait 1 ms tick
- the only stall on a single node
Step 5 · Hand out machine IDs
Assign node bits safely
The whole scheme breaks if two generators are accidentally given the same machine ID — then they collide every millisecond. So who assigns those 1,024 ids, and how do you avoid duplicates as machines come and go?
The scheme breaks if two generators get the same machine id. Who assigns the 1,024 ids?
Random picks collide (birthday paradox at 1,024 slots), and two colliding generators then clash every millisecond. Assignment must guarantee uniqueness, not gamble on it.
Each generator claims a free id once on startup and holds it via a lease. That’s the only coordination in the whole system — after it, every node runs independently and collision-free.
Static config works until autoscaling, re-IPing or cloning silently duplicates an id. A lease-based registry handles machines coming and going without accidental reuse.
Use an ID Registry: on startup each generator claims a free machine id from a coordination service (ZooKeeper/etcd) or static config, holding it via a lease. The only coordination in the entire system happens here, once per process.
Why this piece earns its place
The dangerous moment for a lease is not the registry going down — it is the registry deciding a generator is dead while it is still alive. A long garbage-collection pause or a partition can hold a process off the network past its renewal, the registry hands its slot to a starting instance, and now two live generators own the same slice of every millisecond and clash continuously for as long as both run. So I would deliberately bias reclamation the safe way: a generous lease, a cooling-off period before a released slot returns to the pool, and no reuse of the id of a process that merely stopped renewing. The asymmetry justifies it — a slot you leak costs capacity out of the 1,024 and surfaces as a generator that cannot start, which is loud and fixable; a slot you hand out twice costs uniqueness, which is silent and permanent. Better still, derive the id from a stable identity the orchestrator already assigns, so a restarting process deterministically gets its own slot back rather than racing for a new one.
What the new pieces do
- ID Registryservice
- Hands each generator a unique machine id on startup (via ZooKeeper/etcd or config), so no two instances ever share the same node bits.
Step 6 · When the clock lies
Clock skew & rewind
IDs trust the wall clock — but NTP can step the clock backward. If time moves back, a machine could reuse timestamps it already used and mint duplicate IDs. The one assumption Snowflake makes is the one thing clocks violate.
NTP can step the wall clock backward. What stops the generator minting duplicate IDs?
Rare isn’t never: one NTP correction can reuse a whole millisecond range of IDs, producing duplicates — the one failure this system must never have.
Time zones aren’t the issue — NTP corrections step the underlying wall clock backward regardless of zone. You must detect and guard rewind, not relabel the clock.
A clock guard remembers the last value used and stalls/refuses until the clock passes it — never issuing an ID for a time it already served. A tiny rare pause buys an ironclad no-duplicates guarantee.
A Clock Guard tracks the last timestamp used. If the clock goes backward, the generator refuses to mint (or briefly stalls) until the clock catches up past the last value — never issuing an ID for a time it has already served.
Why this piece earns its place
The guard’s memory lives in the process, and that is the gap worth closing out loud: it knows the last timestamp it served, and a restart wipes that. A machine whose clock steps backward and whose generator restarts a moment later comes up with no record of the future it has already minted into, and every check it makes passes while it re-issues a range it already served. Two things close it. Persist the last timestamp used and refuse to serve on startup until the clock passes it. And configure the host to slew rather than step — drag the clock gradually back into agreement instead of jumping it — so the guard is absorbing drift, which is small, rather than a correction, which need not be. That second one matters because refusing to mint is an outage: a node stalled for the length of a large correction should take itself out of rotation and let callers reach another generator, not hold their requests open.
What the new pieces do
- Clock Guardguard
- Watches for the wall clock jumping backward (NTP correction). If time moves back, it refuses to mint until the clock catches up, preventing duplicate IDs.
Step 7 · Other ways
Alternatives & trade-offs
Snowflake gives unique, k-sorted IDs without coordination — but it leaks creation time (a privacy/competitive signal) and isn’t strictly monotonic across machines. Sometimes you need different properties.
Pick the tool for the job: a DB ticket server or range allocator (hand out blocks of ids per machine) gives strict ordering at the cost of a central component; UUIDv7 offers time-ordered randomness with no machine-id assignment. Snowflake wins when you want local, fast, sortable, and 64-bit.
Why this piece earns its place
What makes this choice different from the others on this page is that it is close to irreversible. The width and type of an id spread outward — into every index, foreign key, API contract, client library and log format that touches one — so moving from a 64-bit integer to a 128-bit UUIDv7 later is not a change to the generator, it is a migration of every table that stores one. The properties to settle before the first id is persisted are exactly the ones you cannot retrofit: how wide the id is, and whether the leading bits carry time. What stays genuinely reversible is where the coordination lives, because a ticket server or a range allocator can be introduced behind the same 64-bit shape if strict ordering turns out to matter later. So the question I would actually ask is whether the storage layer is index-bound. If it is, the ordering and the narrower id earn the machine-id assignment. If it is not, UUIDv7 buys the same time ordering with no registry to run, and the only cost is bytes.
What the new pieces do
- Ticket Serverstore
- An alternative: a database (or range allocator) that hands out monotonically increasing ids. Simpler, strongly ordered, but a potential bottleneck and SPOF.
You did it
You just designed an ID generator.
Everything you assembled, in order
- IDs are generated locally, in-process, with no per-ID network round-trip.
- A timestamp in the high bits makes 64-bit IDs roughly time-sortable.
- A machine-id field keeps same-millisecond IDs on different nodes disjoint.
- A per-millisecond sequence counter handles bursts (4,096 IDs/ms/node).
- A registry assigns unique machine ids once at startup — the only coordination.
- A clock guard refuses to mint when the wall clock rewinds, avoiding duplicates.
- Ticket servers, range allocators and UUIDv7 are alternatives with other trade-offs.