The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
"Design ChatGPT" — what actually makes it hard?
It’s the most-asked AI system design question for a reason. On paper it’s "a form that calls a model." In production it’s ~800M weekly users, on the order of 2.5 billion prompts a day, and every single reply is generated token-by-token on scarce, expensive GPUs. Three things make it hard: conversations are stateful but models are stateless, GPU capacity is the bottleneck for everything, and users judge you on how fast the first token appears.
Design it in layers: a stateless serving path anyone can scale, durable conversation state on the side, a context builder that reconciles the two every turn, and an inference layer that squeezes every token out of the GPUs — with a deliberate degradation plan for the day traffic outruns capacity.
What the new pieces do
- Userclient
- One of ~800M weekly users. Sends a message in an ongoing conversation and expects the reply to start appearing in about a second.
Step 1 · The skeleton
Edge in front, stateless cells behind
Hundreds of millions of users worldwide, hitting you in bursts. Before any AI happens you need boring excellence: authentication, rate limits, and routing to a nearby region. And a fleet of chat servers that can scale horizontally without caring which user lands where.
A user’s next message arrives. Which server should handle it?
Sticky sessions pin users to machines: hot spots, painful deploys, and a lost conversation every time a server dies. State on the serving path is the enemy at this scale.
Billions of conversations would need billions of homes, most idle. You’d be building a distributed state machine you don’t need.
The chat service is stateless: state lives in a store, and every instance can serve every user. That single decision is what makes the fleet trivially scalable.
A global edge/gateway terminates TLS, authenticates, enforces per-user quotas, and routes to the nearest region. Behind it, the Chat Service is a fleet of identical, stateless instances — any one can serve any request, so scaling is "add more."
Why this piece earns its place
Statelessness makes a cell safe to lose, but a chat turn is not a request — it is a connection held open while an answer is generated, and that is what makes the property awkward in practice. A cell holds nothing durable and still holds work in flight, so a release needs a drain mode: refuse new turns, finish the ones already streaming, then exit. The reconnect path will regenerate whatever you cut, so no conversation is at risk, but regenerating is not free — it re-runs the context builder and pays the prefill again on the GPU fleet. Restart a large share of the fleet at once and a routine release becomes a burst of duplicated inference at the moment you have the fewest cells to absorb it, which is an argument for staggering the roll and capping how many drain together. Those same long-lived connections also mislead the load balancer: request rate and CPU both read a cell nursing slow streams as idle, and the quantity worth balancing on is in-flight generations. At one region and a few thousand users you would skip all of it. What the extra hop buys is that the crash path and the release path become the same path — so the one you exercise on every deploy is the one that has to work when a rack dies.
- ~800Mweekly users
- 2.5Bprompts / day
- any cellserves any user
What the new pieces do
- Edge / Gatewayedge
- Global anycast edge: terminates TLS, authenticates, enforces per-user rate limits and quotas, and routes to the nearest healthy region.
- Chat Servicebackend
- The per-request orchestrator. Loads history, builds the prompt, calls inference, streams tokens back. Holds no state itself, so any instance can serve any user.
Step 2 · Remember everything
The conversation store
Conversations are the product: users scroll years of history, resume any thread, and expect nothing to vanish. That’s billions of conversations, most cold, some scorching hot. Where does this state live?
How do you store billions of conversations?
A single write master melts under this write volume. Nothing about this access pattern needs global joins — it’s all "fetch this one conversation."
Every read and write says which conversation it belongs to, so conversation id is the natural shard key. Recent turns stay hot; year-old threads tier down to cheap storage.
A crash loses a day of conversations, and step 1 just made the servers stateless on purpose. Durable state belongs in a store, not a process.
A Conversation Store sharded by conversation id — the access pattern is always "append a turn / read recent turns of one thread," which shards perfectly. Recent messages sit on hot storage; old threads tier to cold object storage nobody pays premium prices to keep.
Why this piece earns its place
Two decisions here are cheap now and expensive later. The first is that an append has to survive being repeated. The retry path regenerates a turn on a different cell, so the same message can reach the store twice, and an auto-incrementing turn number turns that into a user watching their own question appear in their history twice. Derive the row id from what the caller already holds — the conversation id plus a client-side message id — and the duplicate write lands as a no-op. It is also the argument for a row per turn rather than one growing document per thread: appending to a document makes every write a read-modify-write, and there the same duplicate is a lost update instead of a no-op. The second decision is shape. A list is the obvious model, but users edit a message and regenerate a reply, and the moment they do, a conversation is a tree with one currently-selected path through it. Retrofitting that onto a list is a migration across billions of rows; carrying a parent_id from the start costs one column. Then watch where tiering hurts: resuming a two-year-old thread puts a cold-storage read on the critical path before any prefill begins, so it surfaces in dashboards as slow time-to-first-token and sends you hunting through the GPU fleet for a problem that lives in object storage.
What the new pieces do
- Conversation Storestore
- Durable home of every conversation, sharded by conversation id. Recent turns on hot storage, old conversations tiered to cheap cold storage.
Step 3 · The stateless model
Rebuild the context every turn
Here’s the trap most candidates fall into: the model does not remember your conversation. Every API call starts from nothing. So how does turn 47 know what happened in turns 1–46?
How does the model "remember" earlier turns?
Exactly. Each turn, the context builder reassembles the model’s entire input: system prompt, user memory, and as much recent history as fits. "Memory" is an illusion built by the serving system.
Inference is stateless — weights are frozen and no activations persist between requests. If the history isn’t in the prompt, it never happened.
Training per user would be ruinously expensive, days stale, and a privacy minefield. Context assembly, not weight updates, is how chat products remember.
The Context Builder reconstructs the prompt every single turn: system prompt + long-term user memory + recent turns, packed newest-first until the token budget is spent. Older history gets truncated or summarized. The model is stateless; the system supplies the memory.
Why this piece earns its place
The thing worth saying out loud is that this component’s output is a cache key. Prefix reuse downstream depends on this turn’s prompt beginning with the same bytes as last turn’s, and that demands a determinism ordinary request code never has. A timestamp in the system prompt, memory facts emitted in whatever order a set happened to iterate, an experiment flag rewording an instruction: any one changes byte zero, and the prefix is recomputed for that user every turn, forever. Nothing errors and no latency alarm fires; the bill just goes up. The largest breaker, though, is this step’s own policy — when the budget pushes the oldest turn out, everything behind it shifts and the shared prefix ends where the window now starts. So trim in blocks rather than a turn at a time and the break lands once every few turns instead of on every one. The layering above is the cache-safe order for the same reason: the system prompt changes on release, memory occasionally, history every turn, so what moves most sits last and invalidates least. Summarization is that trade plus a write — recomputing a summary per turn would add a model call to every request, so summaries are computed once and stored beside the turns they stand in for, which quietly makes this component a writer and gives you something to invalidate when a user edits an old message.
- every turnprompt rebuilt from scratch
- token budgethard cap per request
- summarizewhen history outgrows it
What the new pieces do
- Context Builderservice
- Rebuilds the model’s entire input every turn: system prompt + user memory + as much recent history as the token budget allows.
Step 4 · Not every prompt deserves a frontier model
The model router
"What’s 2+2?" and "prove this theorem" cost wildly different amounts to answer well — but naive systems send both to the biggest model. At billions of prompts a day, that’s setting money on fire. How do you serve both cheaply and well?
Billions of daily prompts, models from cheap-and-small to frontier. Who decides which model answers?
Most prompts are short, easy, conversational. Answering them with the frontier model multiplies your GPU bill for quality users can’t perceive.
A tiny, fast classifier routes easy traffic to small models and hard reasoning to the frontier model. Done well, it cuts serving cost by multiples with no visible quality drop.
A power-user feature, not a strategy — most users won’t choose, and the default still has to be routed. The system needs its own judgment.
A Model Router scores each prompt — length, topic, reasoning depth, user tier — and dispatches it to the cheapest model that clears the quality bar, escalating to the frontier model only when needed. Routing thresholds are tuned continuously against eval data.
Why this piece earns its place
Routing per prompt sounds obviously right until you notice it is per prompt, inside a conversation that already has cached state attached to one model. KV state is model-specific, so sending turn 13 somewhere different from turn 12 means the whole history is prefilled again from nothing — a route that looks cheaper per token can cost more than staying put. That is why this component has to be cache-aware: the question is not only which model is capable, it is what leaving costs. Sticky-per-conversation routing with a one-way escalation upward is usually the rule that survives contact with production. Two operational notes. The classifier runs ahead of prefill, so its latency lands in time-to-first-token in full and it has to be small enough that the saving is not handed back at the front door. And record the model actually chosen with the turn, not the tier that was asked for — without it you cannot replay a bad answer, and every quality complaint becomes an argument about which model wrote it.
What the new pieces do
- Model Routerservice
- Classifies each prompt and routes it: easy traffic to a small cheap model, hard reasoning to the frontier model. The single biggest cost lever.
Step 5 · Stop paying for the same tokens
Prompt / prefix caching
Look at what the model actually receives each turn: the same system prompt, the same instructions, the same conversation history as last turn — plus one new message. Recomputing attention over that unchanged prefix, every turn, for every user, is an enormous, pure waste.
Most of each request’s prompt is identical to the previous turn. What do you do with that fact?
Prefill (processing the input) is a huge share of inference compute. Ignoring redundancy here leaves massive capacity on the table.
Response caches barely hit in open-ended chat — nearly every conversation is unique. The redundancy worth exploiting is in the prefix, not the answer.
Prefix/KV caching stores the attention state of unchanged prompt prefixes, so the GPU only processes the new tokens. Cheaper prefill AND a much faster first token.
A Prompt Cache keeps the computed KV state for common prefixes — system prompts shared by everyone, and each conversation’s unchanged history. On a hit, prefill work shrinks to just the new tokens, cutting cost and time-to-first-token dramatically.
Why this piece earns its place
The catch is that a hit is not free to reach. KV blocks are cheapest to reuse on the accelerator that computed them; anywhere else — host memory, a shared KV tier — reusing them means moving them across a link before prefill can skip anything. That is still far cheaper than recomputing the prefix, but it is not nothing, so this cache quietly turns a uniform fleet into one with soft affinity: whatever places a turn now prefers the machine already holding that conversation’s blocks, and has to decide per request whether the preference is worth queueing for a busy slot. Lean on locality and the chattiest threads pile onto the same machines; ignore it and you pay a transfer every turn. The second tension is memory. Blocks held for a future turn occupy space a running generation would otherwise use, so every prefix retained is a request you cannot admit. The shared system prompt is easy — everyone hits it, so it stays pinned. One conversation’s history is a bet that its user replies before that space is worth more to somebody else, which makes eviction something to fit against the gap between turns your telemetry already measures, not a constant somebody picked once. If the traffic is mostly one-shot prompts rather than long threads, pin the shared prefix and skip per-conversation caching entirely.
- same prefixturn after turn
- cache hitprocess only new tokens
- TTFTdrops with prefill
What the new pieces do
- Prompt Cachestore
- Caches the computed (KV) state of shared prefixes — system prompts and unchanged history — so prefill work isn’t redone every turn.
Step 6 · The GPU fleet
Continuous batching, or bankruptcy
A GPU running one user’s request at a time idles most of its silicon — decode is memory-bound, not compute-bound. At this scale, serving requests one-by-one wouldn’t just be slow, it would be financially impossible. How does one GPU serve dozens of users at once?
How does a single GPU serve many users’ generations at once?
That’s millions of GPUs. Decode barely uses the GPU’s compute per request; the whole game is sharing each GPU across many requests.
Continuous batching: every decode step advances dozens of interleaved requests, and the scheduler slots new arrivals into seats as they free up — no waiting for the batch to drain.
Swapping means reloading each request’s KV state — exactly the expensive thing to avoid. Batching runs them simultaneously, not in turns.
The Batch Scheduler packs requests onto the GPU Fleet with continuous batching: each forward pass advances every in-flight request by one token, and newly-arrived requests join the moment a slot frees. The binding constraint is KV-cache memory per GPU — that, not compute, decides how many users fit. Admission control keeps queue depth bounded instead of letting latency grow without limit.
Why this piece earns its place
Admission here is a bet, and that is what makes it unlike every other queue you have built. A web request’s cost is known when it arrives; a generation’s is not, because its KV footprint grows with every token and nobody knows how long the answer runs until it stops. So the scheduler admits on an estimate and must be able to preempt — evicting an in-flight request’s state and recomputing or swapping it back later — when the batch outgrows the memory it was admitted against. Build that early, because the alternative is abandoning a half-written reply. The other thing to know is who pays. A new arrival’s prefill is one large compute burst that stalls the decode step of everything already running, so a user who pastes a long prompt degrades other people’s inter-token latency rather than their own. Per-request dashboards look fine throughout, because the victim and the cause are different requests. Chunked prefill exists for exactly that.
What the new pieces do
- Batch Schedulerservice
- Admission control + batching brain. Packs many users’ requests onto each GPU, joins new requests mid-flight, and bounds queue depth.
- GPU Fleetgpu
- Racks of accelerators running the models. Throughput comes from batching; the scarce resource is KV-cache memory per GPU.
Back of the envelope
- continuous batching
- join and leave the batch mid-flight, no drain
- KV-cache memory
- the real capacity limit per GPU
- admission control
- bound the queue; reject/degrade over timing out
- tensor parallel
- shard big models across GPUs in a node
Step 7 · Feel fast
Stream tokens, watch TTFT
A good answer might take 20 seconds to fully generate. Users will not stare at a spinner for 20 seconds — but they’ll happily read along as the answer appears. What does the response path look like, and what number do you actually optimize?
Generation takes ~20s end-to-end. How do you keep it from feeling slow?
Server-sent events / streaming turns 20 dead seconds into a live, readable answer. Perceived latency collapses to the time the FIRST token takes.
A progress bar is still a wait. Streaming gives users the actual product — words — hundreds of times sooner.
Serving a wrong-then-corrected answer erodes trust and doubles cost. Stream the real answer; speed comes from TTFT, not tricks.
Tokens stream back through the chat service to the client as they’re decoded (SSE/WebSocket), the turn is appended to the conversation store, and every request logs TTFT (time to first token), tokens/sec, cost and quality signals to telemetry — the data that tunes the router, the cache and capacity planning. The headline metric is TTFT: it’s what "fast" means in chat.
Why this piece earns its place
Two things about streaming only appear once real users are on it. The first is cancellation. Someone closes the tab, and unless that disconnect travels back through the chat cell to the scheduler, the GPU keeps generating an answer nobody will read — capacity spent on abandoned work that never shows up as an error anywhere. Treat a client disconnect as a first-class signal that frees a batch slot, and decide deliberately whether a partly streamed reply is persisted; if it is not, a user who scrolls away mid-answer returns to a thread holding their question and no reply. The second is measurement. TTFT collected on the server is not what anyone experienced, because proxies buffer streamed responses and a buffer hides the entire problem — instrument it in the client. And the number is bimodal, not noisy: a prefix hit and a prefix miss are two populations with different physics, so one average sits between them and describes neither. Segment by cache outcome and by model, or the metric that defines the product moves for reasons you cannot attribute.
- TTFTthe number users feel
- tokens/sreading speed or better
- every turnlogged for evals
What the new pieces do
- Telemetry / Evalsbus
- Every request’s latency, token counts, cost and quality signals — the feedback loop for capacity planning, routing thresholds and model evals.
The payoff
You designed ChatGPT
From "a form that calls a model" to a planet-scale chat system: stateless cells behind a global edge, conversations sharded by id, context rebuilt every turn under a token budget, a router picking the cheapest capable model, prefix caching killing redundant prefill, continuous batching squeezing the GPUs, and tokens streaming back the moment they exist.
Now spike the traffic and watch the degradation playbook: queue bounds, silent routing downgrades, cache absorption, and free-tier shedding — the difference between a graceful brownout and a global outage.
Everything you assembled, in order
- Stateless cells — any server serves any user; state lives in stores
- Conversation store — sharded by conversation id, hot/cold tiered
- Context builder — model is stateless — rebuild the prompt every turn
- Model router — cheapest capable model; the #1 cost lever
- Prompt cache — reuse KV state of unchanged prefixes
- Continuous batching — dozens of users per GPU, join mid-flight
- KV-cache memory — the real per-GPU capacity limit
- Stream + TTFT — perceived speed = time to first token