The whole design, in writing
Learn system design by building a URL shortener like Bitly or TinyURL step by step. An interactive guide covering the client–server skeleton, base62 key generation, caching the read-heavy redirect path, splitting read/write services, sharding billions of mappings, async click analytics, and the unhappy paths.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a URL shortener?
Strip away the brand and a URL shortener does one tiny thing: take a long, ugly link and hand back a short one — then, when anyone taps the short one, instantly send them to the original.
Sounds trivial. But "instantly", "anyone", and "never collide" turn it into a real systems problem: how do we mint a unique code for every link, and resolve billions of clicks in single-digit milliseconds without ever sending someone to the wrong page?
What the new pieces do
- Shorten UIclient
- Someone with a long, ugly link. Sends it to the service and gets back a tiny code to share.
- Visitorclient
- Anyone who taps a short link. Expects to land on the original page instantly, with no idea a lookup happened.
Step 1 · The skeleton
Two jobs, one server
There are really only two operations: shorten (give me a long URL, return a short code) and redirect (given a code, send me to the long URL). Both need somewhere to keep the mapping — but where?
Two jobs: shorten (long URL → code) and redirect (code → long URL). What’s the minimal structure?
A client can’t share a mapping it created with everyone else who taps the link, and can’t be trusted to keep it. The mapping must live on a server, in durable storage.
To shorten, the server saves code → long_url; to redirect, it looks the code up and returns a 301/302. The classic client → server → database spine: client asks, server decides, database never forgets.
Packing a full URL into a few characters is impossible (URLs are far longer), and the code would no longer be short. You store the mapping and hand back a tiny key to it.
Put a server in the middle and a database behind it. To shorten, the server saves a row code → long_url. To redirect, it looks the code up and answers with an HTTP 301/302 to the original. This is the client → server → database spine under every app.
Why this piece earns its place
The quiet commitment in this step isn’t the server, it’s permanence. The moment the mapping lives on your box, every code you mint is a liability with no end date: a short link gets printed on a poster, pasted into a PDF, embedded in an email somebody opens four years from now. If the store loses that row the link doesn’t degrade, it 404s — on someone else’s page — and nothing on the internet can reconstruct it from the code alone. So I’d rank durability here above everything that comes later. The cache, the key service and the whole analytics side can each be down and the product still half-works; lose code → url and the product is gone retroactively. Two rules fall straight out of that. Never recycle a code: when a dead link is cleaned up you reclaim the bytes, not the key, or an old poster quietly starts sending people somewhere new. And keep the mapping exportable, because the only honest answer to “what happens if you shut down” is handing the table to whoever takes over.
What the new pieces do
- App Serverbackend
- The single front door for every request. Later it becomes an API Gateway sitting behind a load balancer.
- URL Storestore
- Durable storage for the one thing you must never lose: which short code maps to which long URL.
Step 2 · The short code
How do we mint the code?
Every link needs a unique code like /aZ9k2x. Hashing the URL gives collisions and runs long; pure random risks clashes and forces an "is this taken?" check on every single write. How do you guarantee uniqueness cheaply?
Every link needs a unique short code like /aZ9k2x. How do you mint it?
Truncated hashes collide (different URLs → same prefix), forcing a "taken?" check and retries, and identical URLs map to the same code. You want uniqueness without a lookup.
Random codes force a collision check on every write, and clashes rise as the space fills. Counting guarantees uniqueness with no check at all.
Counter 125 → "cb"; the KGS ensures two servers never mint the same number, so codes are unique by construction — no collision check. 62⁷ ≈ 3.5 trillion codes.
Treat it as a number. Keep a global counter and encode it in base62 (a–z A–Z 0–9): counter 125 becomes "cb". A dedicated Key Generation Service hands out ids — so codes are unique by construction, no collision check needed.
Why this piece earns its place
A counter is unique by construction, and it is just as surely predictable by construction. If 125 encodes to “cb”, then the code a second later is the one issued to whoever shortened next — so anybody can start at the short end of the keyspace and walk it, harvesting every link the service has ever minted. That matters because people treat a short link as if it were a secret: an unlisted draft, a document whose only lock is a URL nobody guesses, a one-off invite. None of that is protected by the original link being long and ugly, and a sequential shortener has just published an index of all of it. The fix keeps the property I actually wanted: run the counter through a reversible keyed permutation over the id space before base62-encoding it, so ids stay dense and collision-free while the emitted codes look nothing like a sequence. It costs one fixed function call, on the rare path. The leak has a commercial edge too — a sequence you can walk is a sequence you can difference, which hands a competitor your creation rate.
- 62symbols
- 7chars
- 3.5Tunique codes
What the new pieces do
- Key Gen Serviceservice
- Hands out unique numbers (or pre-made codes) so two servers never mint the same short code.
Back of the envelope
- 62 symbols × 7 slots
- 62⁷ ≈ 3.5 trillion codes
- counter ⇒ unique by construction
- no collision check on write
- KGS hands out ids
- two servers never mint the same code
Step 3 · The hot path
Reads dwarf writes
People create a link once but click it thousands of times. The redirect path runs maybe 100× more than the write path — and every redirect that hits the disk database adds latency to someone’s tap. How do you keep it sub-millisecond?
A link is created once but clicked thousands of times. How do you keep redirects sub-millisecond?
Replicas add throughput but every redirect still does a full DB lookup for an unchanging mapping. When the same code is hit repeatedly, serve it from memory instead.
A hit returns in <1ms; a miss falls through to the store and warms the cache. Because a mapping never changes, it’s perfectly cacheable — no invalidation, pure upside.
You can’t rely on browser caching, and a permanent (301) redirect browsers cache means you never see the click — bad for analytics. Server-side caching keeps you fast AND in the loop.
Put a cache (Redis) in front of the database. A redirect checks the cache first; on a hit it returns in under a millisecond. On a miss it falls through to the database, then warms the cache. Because a mapping never changes, it’s perfectly cacheable.
Why this piece earns its place
The cache introduces the one number on this page you have to choose rather than derive: how much memory to buy. Clicks are brutally front-loaded — a link is shared, it runs hot for a few days, and then it is dead more or less forever — so nearly all of the hit rate comes from recency, and plain LRU over a modest budget does most of the work. Too small and the working set thrashes: every evicted-then-reclicked code pays a sharded store lookup, and the chaos run prices that shape at +9ms at p99. Too large and you are renting RAM to hold links nobody will open again. What I’d really size against, though, is the restart. A cold cache drops the entire read fleet through to the store at once, so the store has to be able to carry the full redirect rate unaided — that same slower-but-correct mode — or the cache has stopped being a latency layer and become a dependency you didn’t agree to.
- 100 : 1reads : writes
- ~4k/sredirects
- <10msp99 lookup
What the new pieces do
- Cachecache
- In-memory store of the hottest code→url mappings. A redirect that hits here returns in under a millisecond.
Back of the envelope
- ~100:1 reads : writes
- created once, clicked thousands of times
- hit ≈ <1ms RAM
- vs a sharded KV lookup
- immutable mapping ⇒ no invalidation
- caching is pure upside
Step 4 · Don’t fall over
Now serve the whole internet
One server was fine for a demo. Under real traffic it melts, and if it dies the whole service goes dark. Reads and writes also have wildly different shapes — mixing them on one box wastes resources. How do you scale?
One server melts under real traffic, and reads outnumber writes ~100:1. How do you scale?
Vertical scaling hits a ceiling and is still one machine — when it dies, the whole service goes dark. And it can’t shape resources to the 100:1 read/write split.
Sharding the DB (step 5) helps storage, but the single app server is still the bottleneck and single point of failure. You also want to scale reads and writes independently.
Run many small stateless copies behind a load balancer so one dying box never takes the system down, and split into a Shorten (write) and Redirect (read) service so the read fleet — 99% of traffic — scales on its own.
Add a load balancer and run many copies (horizontal scaling). Split the work into a Shorten Service (write) and a Redirect Service (read) so each scales — and fails — on its own. The read fleet, carrying 99% of traffic, can grow independently.
Why this piece earns its place
Splitting the fleets buys independent scaling; it does not by itself buy the failure isolation the split implies, because both services still land on the same store and the same shards. A burst of writes — a bulk import, someone scripting the API — can eat the store’s connection budget, and then redirects, 99% of the traffic, get slower because of the 1%. So the two services get separate connection pools and separate quotas against the store, and I’d settle the shedding order before the incident rather than during it: under pressure you refuse to shorten, never to redirect. A failed POST is a retry the creator sees and repeats; a failed redirect is a broken link on a stranger’s page. The split also changes what “healthy” has to mean to the load balancer. An instance that answers a static health endpoint but has lost its store connection will keep being routed to and fail every redirect it gets, so the check has to do a real lookup — otherwise scaling out just multiplies the number of machines confidently serving nothing.
What the new pieces do
- Shorten Serviceservice
- Handles the rare write: takes a long URL, gets a fresh code, and saves the mapping. Scales on its own.
- Redirect Serviceservice
- Handles the overwhelming majority of traffic: look up a code, return a redirect. The fleet you scale the most.
Step 5 · A mountain of links
Where do billions of rows live?
At ~100M new links a month you cross billions of rows within a few years. No single database holds that comfortably, and one disk can’t serve the read rate on its own. Where do they live?
You cross billions of code→url rows in a few years. Where do they live?
A single table of billions of rows outgrows one machine’s disk and read capacity, and you never need joins or queries — only exact lookups by code. A simpler, shardable store fits better.
The access pattern is a pure lookup by code, so a KV store is ideal; sharding by code-hash spreads billions of rows across machines, each shard small and fast. Random-looking codes spread load evenly — no hotspot.
RAM can’t durably hold billions of mappings, and a cache is for the hot subset, not the system of record. You need a durable sharded store behind the cache.
Use a key-value store — the access pattern is a pure lookup by code — and shard it: split the keyspace across many machines, routing each code to its shard by a hash of the code. Each shard stays small and fast; add more as you grow.
Why this piece earns its place
The arithmetic lands on six to eight shards; the decision that actually matters is the function that maps a code to one of them. Hash the code and take it modulo the shard count and growing from six to seven rehomes nearly every key — a migration you get to run underneath live redirect traffic, with a window where a code exists in two places and the wrong one can answer. So fix the mapping now: hash into a large, constant number of virtual buckets and map buckets onto machines. Adding capacity then moves whole buckets, one at a time, and a bucket in flight is a thing you can reason about and roll back. I’d also be precise about what “random codes spread load evenly” does and doesn’t cover. It is true of bytes and broadly true of traffic in aggregate, but one link going viral is a single key on a single shard, and no hash function splits a key. That case is answered by replicating the shard that holds it, not by choosing a cleverer shard key.
- 100Mnew links / mo
- 12B+rows in 10 yrs
- ~6 TBof mappings
Back of the envelope
- 100M/mo × 12 × 10 yrs = 12B rows
- the row count is the easy part — the bytes are what decide the architecture
- 12B × ~500 B/row ≈ 6 TB of mappings
- too big for one box, which is what forces the next line
- 6 TB ÷ ~1 TB/shard ⇒ ~6–8 shards to start
- sized from the arithmetic, with room to split as it grows
- shard by hash(code)
- random codes spread load evenly
- KV lookup, no joins
- each shard stays small and fast
Step 6 · Count the clicks
Who clicked, from where?
Owners want stats — clicks, countries, referrers, devices. But writing an analytics row on every redirect would slow the one thing that must stay fast: the redirect itself. How do you have both?
Owners want click stats, but writing a row on every redirect would slow it. How?
Blocking the redirect on an analytics write couples the must-be-fast path to a slower one, and an analytics outage would break redirects. Counting must never gate the click.
Owners genuinely want stats, and you can have both — you don’t have to choose. Decouple counting from serving instead of dropping it.
The Redirect Service emits an event to Kafka and returns immediately; downstream consumers fold events into the analytics store at their own pace. If analytics lags or crashes, redirects keep flying.
Make analytics asynchronous. The Redirect Service fires a lightweight event onto a stream (Kafka) and immediately returns the redirect. Downstream consumers fold those events into an analytics store at their own pace — the click is never blocked on counting it.
Why this piece earns its place
Fire-and-forget is a delivery guarantee, and the guarantee is at most once. Clicks will be lost: sitting in a producer buffer when an instance is recycled mid-deploy, and across any window where the stream is unreachable — which is exactly the outage this page lets you trigger. That is the right trade, but it has to be a stated one. The owner-facing figure is an estimate, and if you label it “clicks” with no qualifier, the first customer who diffs it against their own site analytics files a bug you can never close. Call it approximate and reconcile toward the destination’s numbers instead of promising to match them. The implementation detail that decides whether the trade holds at all is what the producer does when its buffer fills. Most clients default to blocking the caller until there is room, which silently turns the decoupled path back into a synchronous one — the redirect now waits on the analytics system, the single thing this step exists to prevent. Configure it to drop, and count the drops, because that counter is your only evidence the loss happened.
What the new pieces do
- Analyticsstore
- Aggregates clicks, countries, referrers and devices — built up asynchronously so it never slows a redirect.
- Click Eventsbus
- A firehose of click events. Lets redirects fire-and-forget while analytics catches up at its own pace.
Step 7 · The sharp edges
Aliases, expiry & abuse
Real users want custom aliases (/launch), links that expire, and — less welcome — spammers who shorten malicious URLs or hammer your API.
Let writers request a custom code (check it’s free first). Store an optional TTL so expired links return 404 and get cleaned up. Add rate limiting at the gateway and a safe-browsing check on new URLs, so one bad actor can’t ruin the service for everyone.
Why this piece earns its place
Screening a URL on write quietly assumes a URL is a fixed thing. It isn’t — the destination is someone else’s server, and the profitable play is to shorten something harmless, pass the check, seed the link, then swap the page for a phishing form a week later once it has spread. So screening can’t only be a gate at creation. It has to re-check links as they get popular, which the click stream from the previous step already tells you about for free, alongside a report path and a way to kill a code on the spot. Killing it is where this design needs the one thing it has so far been able to avoid. The mapping is write-once, which is precisely why there is no invalidation anywhere in the caching story — but a takedown is a deletion, and deletion is the write the cache never hears about. Remove the row and a flagged link keeps redirecting from memory until the entry happens to age out. The kill path has to reach the cache explicitly. It is also the sharpest argument for the 302: a 301 you handed out months ago lives in browsers you cannot reach, and can never be recalled.
You did it
You just designed a URL shortener.
Everything you assembled, in order
- Client → server → database — the spine shared by shorten and redirect.
- Base62 codes from a Key Gen Service — unique by construction.
- A Redis cache makes the read-heavy redirect path sub-millisecond.
- Load balancer + split read/write services to scale horizontally.
- A key-value store sharded by code holds billions of mappings.
- Async click events via Kafka — analytics never blocks a redirect.
- Custom aliases, TTL expiry, 404s and rate limiting for the real world.