Design Pastebin

The big idea

What is Pastebin?

Paste some text, get a short link, share it; anyone who opens the link sees the text. Dead simple — but at scale it’s a sharp lesson in read-heavy design: pastes are written once and read thousands of times, sometimes from all over the world.

Mint a unique short key, store the (potentially large) text in a blob store with only a pointer in the database, and make the read path fly with caching and a CDN. It’s the URL shortener’s cousin — but with real content to store and expire.

How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.

Step 1 · The skeleton

Store text, return a key

Two operations: create (take text, give back a key) and read (take a key, give back text). A paste can be a few bytes or a few megabytes, so where the text lives matters from the start.

Design decision: A paste can be a few bytes or a few megabytes. Where do you put the text?

The call: Text in a blob/object store; a small pointer row in the DB. — The DB holds tiny queryable rows (key → location, TTL, flags) while the object store holds the bulk. Same split as photos/videos — small fast database, bulk where bulk belongs.

The API Server writes the text to an Object Store and a small row to the Metadata DB mapping the key to that text’s location. Reads resolve the key in the DB, then fetch the body from the blob store.

Separate metadata from content: Big bodies in a blob store, tiny queryable rows in the DB. The same split as photos or videos — keep the database small and fast, keep the bulk where bulk belongs.

Step 2 · The short link

Mint a unique key

Every paste needs a unique, short, URL-safe code like /aZ9k2. Hashing the content collides on identical pastes and yields long codes; pure random forces a “taken?” check on every write.

Design decision: Every paste needs a unique short URL-safe code like /aZ9k2. How do you mint it without collisions?

The call: Encode an incrementing counter in base62. — Counting up then base62-encoding guarantees no collisions without any lookup; ~7 chars covers trillions of pastes. (Or hand out pre-generated unused keys.) Unique by construction.

Use a Key Gen Service: take a counter and encode it in base62, or hand out pre-generated unused keys. Codes are unique by construction, so creating a paste never needs a collision check.

Uniqueness by construction: Counting up (then base62-encoding) guarantees no collisions without a lookup. A handful of base62 characters covers trillions of pastes — you’ll never run out.

Step 3 · Reads dwarf writes

Cache hot pastes

A paste shared in a popular thread gets read thousands of times. Hitting the metadata DB and blob store for every one of those identical reads is wasteful and slow.

Design decision: A paste in a popular thread is read thousands of times. How do you avoid hitting the DB + blob store every time?

The call: Cache hot pastes in Redis, keyed by paste id. — A hit returns the text in <1ms; a miss falls through to the blob store and warms the cache. Pastes are write-once, so there’s no invalidation — caching is pure upside from the second read on.

Put a Cache (Redis) on the read path keyed by paste id. A hit returns the text in under a millisecond; a miss falls through to the blob store and warms the cache. Because a paste’s text never changes, it’s perfectly cacheable.

Immutable = cache-friendly: Pastes are write-once. No edits means no invalidation, so caching is pure upside — the second read onward is essentially free.

Step 4 · Readers everywhere

Go global with a CDN

Your cache and store sit in one region, but readers are worldwide. A paste opened from another continent pays a long round-trip on every fetch.

Design decision: Your cache and store sit in one region but readers are worldwide. How do you cut the cross-continent round-trip?

The call: Front the read path with a CDN; immutable content + long TTL. — The first reader in a region pulls through the edge; everyone after is served locally. Immutable pastes cache at the edge with long TTLs — the cheapest scaling for read-heavy payloads.

Front the read path with a CDN. The first reader in a region pulls the paste through the edge; everyone after is served locally. Immutable content plus a long TTL means the edge does the heavy lifting.

Edge-cache the content: For read-heavy, immutable payloads, a CDN is the cheapest scaling you can buy — it moves both latency and bandwidth off your origin and out to the edge.

Step 5 · Don’t keep it forever

Expiration & TTL

Many pastes are meant to be temporary (“expire in 1 hour / 1 day”), and keeping every paste forever steadily fills storage with dead content nobody will ever read again.

Design decision: Many pastes are meant to be temporary ("expire in 1 hour"). How do you stop storage filling with dead content?

The call: Store a TTL per paste; lazy-delete on read + a background sweep. — An expiry on the record makes cleanup routine: expired keys return 404/410, lazy deletion drops them on access, and a background worker (or native store TTL) sweeps the rest. Lifetime becomes data.

Store an optional TTL with each paste. An Expiry Worker (or the store’s native TTL) removes expired pastes from the blob store and metadata, after which the key returns a clean 404/410.

Lifetime as data: Attaching an expiry to the record turns cleanup into routine maintenance instead of a guess. Lazy deletion on read plus a background sweep keeps storage honest.

Step 6 · Who can see it?

Privacy & limits

Not every paste should be public. Some are unlisted (only those with the link) or private (only the author). And without size limits, one user can paste a gigabyte and wreck your storage and bandwidth.

Design decision: Some pastes should be unlisted (link-only) or private (author-only). How do you enforce that?

The call: A privacy flag enforced at the API on every read (+ auth for private). — Store a per-paste flag; public serves freely, unlisted needs the key, private requires the author’s auth — checked on every read. Pair with size caps and rate limits so abuse can’t balloon costs.

Add a privacy flag per paste, enforced at the API on every read, and require the right key/auth for private ones. Cap paste size and rate-limit creation so abuse can’t balloon costs or hammer the service.

Design for the unhappy path: Unlisted ≠ secure, so private pastes need real access checks, not just an obscure URL. Size caps and rate limits are the boring guardrails that keep a free service alive.

Step 7 · Counts & scale

Analytics, async

Owners want view counts, but writing an analytics row on every read would slow the one thing that must stay fast — and billions of small rows pile up in one database over time.

Record views asynchronously (fire an event, aggregate later) so reads never block on counting. Shard the Metadata DB by key and lean on the cache/CDN so the read path stays flat as paste count grows into the billions.

Never block the hot path: Counting is best-effort and can lag; serving the paste cannot. Push analytics off the request, shard the metadata, and let the edge absorb the reads.

You did it

You just designed Pastebin.

• Create/read over an API, with text in a blob store and a pointer in the DB.
• Base62 keys from a Key Gen Service — unique by construction.
• A Redis cache makes the read-heavy, immutable path sub-millisecond.
• A CDN serves pastes from the edge for global readers.
• TTL-based expiry (lazy delete + sweep) keeps storage from filling with dead pastes.
• Per-paste privacy flags, size caps and rate limits for the real world.
• Async analytics and a sharded metadata DB keep the read path flat at scale.