The big idea

"Design ChatGPT" — what actually makes it hard?

It’s the most-asked AI system design question for a reason. On paper it’s "a form that calls a model." In production it’s ~800M weekly users, on the order of 2.5 billion prompts a day, and every single reply is generated token-by-token on scarce, expensive GPUs. Three things make it hard: conversations are stateful but models are stateless, GPU capacity is the bottleneck for everything, and users judge you on how fast the first token appears.

Design it in layers: a stateless serving path anyone can scale, durable conversation state on the side, a context builder that reconciles the two every turn, and an inference layer that squeezes every token out of the GPUs — with a deliberate degradation plan for the day traffic outruns capacity.

How to read this: Each step opens with a real design decision — make the call before seeing what ships. Watch the system grow, hover any box, replay the flow. At the end, spike the traffic and watch the load-shedding playbook fire. Hit Begin.

Step 1 · The skeleton

Edge in front, stateless cells behind

Hundreds of millions of users worldwide, hitting you in bursts. Before any AI happens you need boring excellence: authentication, rate limits, and routing to a nearby region. And a fleet of chat servers that can scale horizontally without caring which user lands where.

Design decision: A user’s next message arrives. Which server should handle it?

The call: Any server — because no server holds conversation state. — The chat service is stateless: state lives in a store, and every instance can serve every user. That single decision is what makes the fleet trivially scalable.

A global edge/gateway terminates TLS, authenticates, enforces per-user quotas, and routes to the nearest region. Behind it, the Chat Service is a fleet of identical, stateless instances — any one can serve any request, so scaling is "add more."

Stateless serving path: Push all state off the request path and into stores built for it. Stateless services scale linearly, deploy safely, and fail without losing anything a retry can’t recover.

Step 2 · Remember everything

The conversation store

Conversations are the product: users scroll years of history, resume any thread, and expect nothing to vanish. That’s billions of conversations, most cold, some scorching hot. Where does this state live?

Design decision: How do you store billions of conversations?

The call: A horizontally sharded store, keyed by conversation id, with hot/cold tiers. — Every read and write says which conversation it belongs to, so conversation id is the natural shard key. Recent turns stay hot; year-old threads tier down to cheap storage.

A Conversation Store sharded by conversation id — the access pattern is always "append a turn / read recent turns of one thread," which shards perfectly. Recent messages sit on hot storage; old threads tier to cold object storage nobody pays premium prices to keep.

Shard on the access pattern: Pick the shard key from how data is actually read. Chat is always per-conversation, so conversation id gives single-shard reads and writes — no cross-shard queries, no hot master.

Step 3 · The stateless model

Rebuild the context every turn

Here’s the trap most candidates fall into: the model does not remember your conversation. Every API call starts from nothing. So how does turn 47 know what happened in turns 1–46?

Design decision: How does the model "remember" earlier turns?

The call: It doesn’t — the system re-sends relevant history inside every request. — Exactly. Each turn, the context builder reassembles the model’s entire input: system prompt, user memory, and as much recent history as fits. "Memory" is an illusion built by the serving system.

The Context Builder reconstructs the prompt every single turn: system prompt + long-term user memory + recent turns, packed newest-first until the token budget is spent. Older history gets truncated or summarized. The model is stateless; the system supplies the memory.

Context is a budget, not a bucket: The context window is finite and every token costs prefill compute. Spend it deliberately: pin what must be there (system prompt, memory), rank the rest, cut the tail. This is context engineering in one sentence.

Step 4 · Not every prompt deserves a frontier model

The model router

"What’s 2+2?" and "prove this theorem" cost wildly different amounts to answer well — but naive systems send both to the biggest model. At billions of prompts a day, that’s setting money on fire. How do you serve both cheaply and well?

Design decision: Billions of daily prompts, models from cheap-and-small to frontier. Who decides which model answers?

The call: A router classifies each prompt and picks the cheapest model that will answer it well. — A tiny, fast classifier routes easy traffic to small models and hard reasoning to the frontier model. Done well, it cuts serving cost by multiples with no visible quality drop.

A Model Router scores each prompt — length, topic, reasoning depth, user tier — and dispatches it to the cheapest model that clears the quality bar, escalating to the frontier model only when needed. Routing thresholds are tuned continuously against eval data.

The router is the cost lever: Serving cost is dominated by which model runs. A router that sends even half the traffic to a model 10× cheaper roughly halves the entire GPU bill — no other single component comes close.

Step 5 · Stop paying for the same tokens

Prompt / prefix caching

Look at what the model actually receives each turn: the same system prompt, the same instructions, the same conversation history as last turn — plus one new message. Recomputing attention over that unchanged prefix, every turn, for every user, is an enormous, pure waste.

Design decision: Most of each request’s prompt is identical to the previous turn. What do you do with that fact?

The call: Cache the computed KV state of shared prefixes and reuse it. — Prefix/KV caching stores the attention state of unchanged prompt prefixes, so the GPU only processes the new tokens. Cheaper prefill AND a much faster first token.

A Prompt Cache keeps the computed KV state for common prefixes — system prompts shared by everyone, and each conversation’s unchanged history. On a hit, prefill work shrinks to just the new tokens, cutting cost and time-to-first-token dramatically.

Prefill vs decode: Inference has two phases: prefill (read the prompt, build KV state) and decode (emit tokens one by one). Prefix caching attacks prefill; batching (next step) attacks decode. You need both.

Step 6 · The GPU fleet

Continuous batching, or bankruptcy

A GPU running one user’s request at a time idles most of its silicon — decode is memory-bound, not compute-bound. At this scale, serving requests one-by-one wouldn’t just be slow, it would be financially impossible. How does one GPU serve dozens of users at once?

Design decision: How does a single GPU serve many users’ generations at once?

The call: Batch many requests per forward pass, joining new ones as others finish mid-flight. — Continuous batching: every decode step advances dozens of interleaved requests, and the scheduler slots new arrivals into seats as they free up — no waiting for the batch to drain.

The Batch Scheduler packs requests onto the GPU Fleet with continuous batching: each forward pass advances every in-flight request by one token, and newly-arrived requests join the moment a slot frees. The binding constraint is KV-cache memory per GPU — that, not compute, decides how many users fit. Admission control keeps queue depth bounded instead of letting latency grow without limit.

Throughput lives in the batch: Per-request, decode is memory-bandwidth-bound and wastes the GPU. Batched, the same pass serves dozens of requests almost for free. Tokens/sec per GPU — and therefore cost per token — is set by how well you batch.

Step 7 · Feel fast

Stream tokens, watch TTFT

A good answer might take 20 seconds to fully generate. Users will not stare at a spinner for 20 seconds — but they’ll happily read along as the answer appears. What does the response path look like, and what number do you actually optimize?

Design decision: Generation takes ~20s end-to-end. How do you keep it from feeling slow?

The call: Stream each token to the client the moment it’s generated. — Server-sent events / streaming turns 20 dead seconds into a live, readable answer. Perceived latency collapses to the time the FIRST token takes.

Tokens stream back through the chat service to the client as they’re decoded (SSE/WebSocket), the turn is appended to the conversation store, and every request logs TTFT (time to first token), tokens/sec, cost and quality signals to telemetry — the data that tunes the router, the cache and capacity planning. The headline metric is TTFT: it’s what "fast" means in chat.

Optimize perceived latency: Users experience two numbers: how fast the answer starts (TTFT — prefill + queueing) and how fast it flows (tokens/sec — decode). Prefix caching and admission control buy the first; batching efficiency buys the second.

The payoff

You designed ChatGPT

From "a form that calls a model" to a planet-scale chat system: stateless cells behind a global edge, conversations sharded by id, context rebuilt every turn under a token budget, a router picking the cheapest capable model, prefix caching killing redundant prefill, continuous batching squeezing the GPUs, and tokens streaming back the moment they exist.

Now spike the traffic and watch the degradation playbook: queue bounds, silent routing downgrades, cache absorption, and free-tier shedding — the difference between a graceful brownout and a global outage.

• Stateless cells — any server serves any user; state lives in stores
• Conversation store — sharded by conversation id, hot/cold tiered
• Context builder — model is stateless — rebuild the prompt every turn
• Model router — cheapest capable model; the #1 cost lever
• Prompt cache — reuse KV state of unchanged prefixes
• Continuous batching — dozens of users per GPU, join mid-flight
• KV-cache memory — the real per-GPU capacity limit
• Stream + TTFT — perceived speed = time to first token