The whole design, in writing
Learn system design by building a text-paste service like Pastebin step by step. An interactive guide covering unique key generation, separating metadata from blob storage, caching and CDN for read-heavy traffic, TTL expiration, privacy, and async analytics.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is Pastebin?
Paste some text, get a short link, share it; anyone who opens the link sees the text. Dead simple — but at scale it’s a sharp lesson in read-heavy design: pastes are written once and read thousands of times, sometimes from all over the world.
Mint a unique short key, store the (potentially large) text in a blob store with only a pointer in the database, and make the read path fly with caching and a CDN. It’s the URL shortener’s cousin — but with real content to store and expire.
What the new pieces do
- Creatorclient
- Pastes some text — a log, a snippet, an error — and gets back a short URL to share. Writes are rare compared to reads.
- Readerclient
- Opens a shared paste link. Expects the text to load instantly, possibly thousands of times for one popular paste.
Step 1 · The skeleton
Store text, return a key
Two operations: create (take text, give back a key) and read (take a key, give back text). A paste can be a few bytes or a few megabytes, so where the text lives matters from the start.
A paste can be a few bytes or a few megabytes. Where do you put the text?
Multi-megabyte bodies bloat the DB, slow every query, and waste its indexed storage on data you only fetch by key. Keep big bodies out of the database.
The DB holds tiny queryable rows (key → location, TTL, flags) while the object store holds the bulk. Same split as photos/videos — small fast database, bulk where bulk belongs.
In-memory pastes vanish on restart and can’t scale past one machine’s RAM. Pastes must be durably stored, with memory used only as a cache later.
The API Server writes the text to an Object Store and a small row to the Metadata DB mapping the key to that text’s location. Reads resolve the key in the DB, then fetch the body from the blob store.
What the new pieces do
- API Serverbackend
- Handles both operations: store a new paste and return its key, or look up a key and return the text. The single front door.
- Metadata DBstore
- Small rows: the key, where the text lives, expiry time, and privacy flag. Queried on every read to resolve a key to its blob.
- Object Storestore
- The actual paste bodies. Kept out of the database so large pastes don’t bloat it; the DB holds only a pointer here.
Step 2 · The short link
Mint a unique key
Every paste needs a unique, short, URL-safe code like /aZ9k2. Hashing the content collides on identical pastes and yields long codes; pure random forces a “taken?” check on every write.
Every paste needs a unique short URL-safe code like /aZ9k2. How do you mint it without collisions?
Hashing collides on identical pastes (two people paste the same snippet → same key, overwriting), and full hashes are long. You need short codes unique regardless of content.
Random codes force a "is it taken?" lookup on every create, and collisions rise as you fill the space. Uniqueness by construction avoids the check entirely.
Counting up then base62-encoding guarantees no collisions without any lookup; ~7 chars covers trillions of pastes. (Or hand out pre-generated unused keys.) Unique by construction.
Use a Key Gen Service: take a counter and encode it in base62, or hand out pre-generated unused keys. Codes are unique by construction, so creating a paste never needs a collision check.
- base62encoding
- ~7chars
- trillionsof keys
What the new pieces do
- Key Genservice
- Mints a unique short code (base62) for each paste so two pastes never collide — uniqueness by construction, no “is it taken?” check.
Back of the envelope
- base62, ~7 chars
- 62^7 ≈ 3.5 trillion keys
- counter ⇒ unique by construction
- no collision check on create
- short + URL-safe
- a–z A–Z 0–9, no escaping
Step 3 · Reads dwarf writes
Cache hot pastes
A paste shared in a popular thread gets read thousands of times. Hitting the metadata DB and blob store for every one of those identical reads is wasteful and slow.
A paste in a popular thread is read thousands of times. How do you avoid hitting the DB + blob store every time?
Replicas add throughput but every hot read still does a full store fetch for identical, unchanging content. When the same paste repeats, serve it from memory instead of re-fetching.
A hit returns the text in <1ms; a miss falls through to the blob store and warms the cache. Pastes are write-once, so there’s no invalidation — caching is pure upside from the second read on.
You can’t control reader behavior, and the goal is to serve many readers fast. The system must make repeat reads cheap server-side via caching, not push work onto users.
Put a Cache (Redis) on the read path keyed by paste id. A hit returns the text in under a millisecond; a miss falls through to the blob store and warms the cache. Because a paste’s text never changes, it’s perfectly cacheable.
- <1mscache hit
- 100:1reads : writes
- noinvalidation
What the new pieces do
- Cachecache
- Redis holding the most-read pastes. A popular link is served from memory in under a millisecond instead of fetching the blob every time.
Back of the envelope
- ~100:1 reads : writes
- write once, read thousands of times
- hit ≈ <1ms RAM
- vs a DB lookup + blob fetch
- immutable ⇒ no invalidation
- caching is pure upside
Step 4 · Readers everywhere
Go global with a CDN
Your cache and store sit in one region, but readers are worldwide. A paste opened from another continent pays a long round-trip on every fetch.
Your cache and store sit in one region but readers are worldwide. How do you cut the cross-continent round-trip?
More nodes in one region don’t help a reader on another continent — they still pay the long round-trip on every fetch. You need content closer to the reader, geographically.
Hand-rolling global replication of all content is expensive and heavy when most pastes are never read in most regions. A CDN already does edge caching on demand.
The first reader in a region pulls through the edge; everyone after is served locally. Immutable pastes cache at the edge with long TTLs — the cheapest scaling for read-heavy payloads.
Front the read path with a CDN. The first reader in a region pulls the paste through the edge; everyone after is served locally. Immutable content plus a long TTL means the edge does the heavy lifting.
What the new pieces do
- CDNcache
- Caches paste content near readers worldwide. Since a paste’s text is immutable, it can sit at the edge with a long TTL.
Back of the envelope
- 1st regional read pulls through edge
- every reader after is served locally
- immutable + long TTL
- the edge does the heavy lifting
- offloads latency + bandwidth
- origin barely touched
Step 5 · Don’t keep it forever
Expiration & TTL
Many pastes are meant to be temporary (“expire in 1 hour / 1 day”), and keeping every paste forever steadily fills storage with dead content nobody will ever read again.
Many pastes are meant to be temporary ("expire in 1 hour"). How do you stop storage filling with dead content?
An expiry on the record makes cleanup routine: expired keys return 404/410, lazy deletion drops them on access, and a background worker (or native store TTL) sweeps the rest. Lifetime becomes data.
Manual cleanup doesn’t scale and risks deleting the wrong thing under pressure. Expiry should be declared per paste and enforced automatically, not firefought.
Keeping every temporary paste forever fills storage with content nobody will read again, and ignores users who explicitly wanted expiry. TTL is both a feature and a cost control.
Store an optional TTL with each paste. An Expiry Worker (or the store’s native TTL) removes expired pastes from the blob store and metadata, after which the key returns a clean 404/410.
What the new pieces do
- Expiry Workerworker
- Removes pastes whose TTL has passed from the store and metadata, reclaiming space so expired links return 404.
Step 6 · Who can see it?
Privacy & limits
Not every paste should be public. Some are unlisted (only those with the link) or private (only the author). And without size limits, one user can paste a gigabyte and wreck your storage and bandwidth.
Some pastes should be unlisted (link-only) or private (author-only). How do you enforce that?
An obscure URL is "security by obscurity" — unlisted is not private. Links leak, get shared and indexed, so true privacy needs a real access check, not just a random key.
Hiding from a listing does nothing if anyone with the key can still GET the content. The read path itself must verify permission, not just the discovery surface.
Store a per-paste flag; public serves freely, unlisted needs the key, private requires the author’s auth — checked on every read. Pair with size caps and rate limits so abuse can’t balloon costs.
Add a privacy flag per paste, enforced at the API on every read, and require the right key/auth for private ones. Cap paste size and rate-limit creation so abuse can’t balloon costs or hammer the service.
Step 7 · Counts & scale
Analytics, async
Owners want view counts, but writing an analytics row on every read would slow the one thing that must stay fast — and billions of small rows pile up in one database over time.
Record views asynchronously (fire an event, aggregate later) so reads never block on counting. Shard the Metadata DB by key and lean on the cache/CDN so the read path stays flat as paste count grows into the billions.
What the new pieces do
- Analyticsstore
- Tracks how often each paste is viewed, updated asynchronously so counting a view never slows the read itself.
You did it
You just designed Pastebin.
Everything you assembled, in order
- Create/read over an API, with text in a blob store and a pointer in the DB.
- Base62 keys from a Key Gen Service — unique by construction.
- A Redis cache makes the read-heavy, immutable path sub-millisecond.
- A CDN serves pastes from the edge for global readers.
- TTL-based expiry (lazy delete + sweep) keeps storage from filling with dead pastes.
- Per-paste privacy flags, size caps and rate limits for the real world.
- Async analytics and a sharded metadata DB keep the read path flat at scale.