The whole design, in writing
Learn AI system design by building a production conversational AI like ChatGPT step by step. An interactive guide covering the inference gateway, context-window assembly and token budgeting, conversation memory, model serving with batching and KV cache, token streaming, and prompt/output guardrails.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a conversational AI, really?
Strip away the chat bubble and a system like ChatGPT does one deceptively simple thing: take your message plus everything said so far, run it through a language model, and stream back an answer — for millions of people at once.
That sentence hides the hard parts. What exactly do we feed the model (and how much fits)? How does it remember earlier turns? How do we serve a giant model fast and affordably? And how do we keep the output safe?
What the new pieces do
- Userclient
- A person typing a message. Sends a turn, then watches the answer stream back token by token.
Step 1 · The skeleton
A client, a gateway, a model
Your browser can’t hold a 100-billion-parameter model, and it can’t be trusted with the keys, the rate limits, or other users’ data. So where does the request go?
You type a message. Where does the thinking happen?
Frontier models are tens to hundreds of gigabytes and need GPUs. Even if it fit, you’d ship the weights to every user and lose all control over cost and safety.
A neutral backend holds the keys, enforces limits, assembles the prompt and routes to the GPU fleet. This is the spine of every LLM product.
Letting clients hit inference directly means no auth boundary, no rate limiting, no prompt assembly, and your model endpoint exposed to the world.
We put an Inference Gateway in the middle. The browser sends a turn; the gateway authenticates, rate-limits, and orchestrates everything behind it. This is the client → gateway → model backbone under every chat product.
What the new pieces do
- Inference Gatewaybackend
- The single front door for every chat request. Authenticates, rate-limits, then orchestrates context, guardrails and model serving.
Step 2 · What the model sees
Assemble the context window
A language model has no memory between calls — it only sees the text you hand it this request. So what exactly do we put in front of it for each turn?
The model is stateless. What do you send it for each new message?
Then it forgets the system instructions and everything said earlier — every turn starts from scratch. The "conversation" disappears.
Right instinct, wrong limit. Conversations grow past the context window and cost scales with every token sent. You need to assemble and budget.
A Context Builder composes the model input each turn — the system prompt, the recent/relevant history, and the new message — shaped to fit the window.
A Context Builder composes the model input for every turn: the system prompt (who the assistant is), the conversation history, and the new message. The model is stateless, so the full context is rebuilt and resent on each request.
What the new pieces do
- Context Builderservice
- Assembles the model input: system prompt + recent history + the new message, trimmed to fit the context window.
Step 3 · Budget the tokens
It won’t all fit — now what?
The window is finite (say 128k tokens) and every token costs money and latency. A long chat — or a big document — blows past it. How do you decide what makes the cut?
The conversation is longer than the context window. What gets sent?
Overflowing the window errors or silently drops the start — usually the system prompt, the most important part. You must decide what to keep on purpose.
Allocate the token budget deliberately — pin the system prompt, keep recent turns, compress older ones, and pull in only the relevant facts (RAG) instead of everything.
Recency usually matters most in a conversation, and the oldest turns are often stale. Blindly keeping the start wastes budget on what no longer matters.
The builder works to a token budget. It pins the system prompt, keeps the most recent turns, and summarizes or drops older ones. For knowledge beyond the chat, it retrieves only the relevant passages from a vector store (RAG) instead of stuffing everything in.
- 128kwindow (tokens)
- ~4 chars≈ 1 token
- in + outboth billed
What the new pieces do
- Vector / KBindex
- Optional knowledge source. The context builder can retrieve relevant facts to ground the answer (RAG).
Back of the envelope
- system prompt
- pinned — always sent
- recent turns
- kept verbatim while they fit
- older turns
- summarized into a running recap
- external knowledge
- retrieved on demand (RAG), not preloaded
Step 4 · Make it remember
Where does history live?
The model is stateless and the browser can’t be trusted to hold the true history. Yet tomorrow the user reopens the chat and expects it all to be there. Where does the conversation actually live?
The model forgets between calls. Where is the real conversation stored?
Inference doesn’t update the weights. The model never remembers anything between requests; memory must live outside it.
Close the tab and it’s gone; switch devices and it’s gone; you can’t rebuild context server-side. The client is a cache, not the source of truth.
Persist every turn server-side. The builder loads history from it each request and saves the new turn — survives reloads, devices and crashes.
A Memory Service reads and writes the Conversation DB — the durable source of truth for every turn. Each request, the builder loads recent history from it; after each answer, the new turn is appended. The browser is just a fast cache of what the database already knows.
What the new pieces do
- Memory Serviceservice
- Loads and saves conversation history so a chat remembers earlier turns across requests.
- Conversation DBstore
- Durable storage of every message in every conversation — the source of truth for history.
Step 5 · The real cost
Inference is the expensive part
Assembling a prompt is cheap. Running a hundred-billion-parameter model to generate each token is not — it needs GPUs, and a single request can hold one for seconds. How should the system treat this?
What dominates the cost and latency of a chat turn?
A few small reads are microseconds. Real, but a rounding error next to generation.
Each output token is a full forward pass through the model on scarce GPUs. This dominates both cost and latency, so it gets its own carefully-managed tier.
Bytes are tiny; latency here is milliseconds. The seconds you wait are the model thinking, not the wire.
Generation runs on a dedicated Model Servers tier — a fleet of GPUs holding the model weights. Because each output token is a full pass through the model, this tier is the bottleneck we design everything else around: it gets its own queue, its own scaling, and its own failure handling.
Step 6 · Serve it efficiently
Batching and the KV cache
GPUs are most efficient when fully fed, but chat requests arrive one at a time and each generates tokens one at a time. Run them naively and the GPUs sit half-idle while users wait. How do you serve a giant model fast?
How do you keep scarce GPUs busy across many concurrent chats?
A single chat can’t saturate a GPU, and others queue behind it. You pay for hardware that mostly idles while users wait.
Pack concurrent requests into shared batches and cache the attention state (KV cache) so each new token is cheap. This is how real serving stacks hit high throughput.
Re-running attention over the entire context for each output token is quadratic waste — exactly what the KV cache exists to avoid.
The model servers use continuous batching — packing many in-flight chats into shared GPU passes — and a KV cache that stores attention state so each new token reuses prior work instead of recomputing the whole prompt. Every turn also emits usage events (tokens, latency) to the stream for billing and evals.
What the new pieces do
- Model Serversservice
- The GPU fleet running the LLM. Batches concurrent requests and reuses the KV cache to generate tokens efficiently.
- Model Weightsstore
- The model parameters, loaded across the GPU fleet. Large models are sharded over many GPUs.
- Event Streambus
- Every turn emits events — tokens used, latency, safety flags — feeding billing, evals and abuse detection.
Back of the envelope
- continuous batching
- merge concurrent requests into shared GPU passes
- KV cache
- reuse attention state — no recompute per token
- weights sharded across GPUs
- big models span many devices
- emit usage events
- tokens + latency → billing, evals, abuse detection
Step 7 · Don’t make them wait
Stream the tokens back
A long answer can take many seconds to finish. If the user stares at a blank screen until it’s done, the product feels broken — even when it’s working perfectly. How do you make it feel instant?
A full answer takes 6 seconds to generate. What does the user see?
Six seconds of blank screen reads as "frozen." You’re hiding progress the model is already making.
Stream over a persistent connection (SSE/WebSocket) so words appear as they’re produced. Time-to-first-token is what users actually feel.
Polling adds latency and load and still lags the model. A push stream delivers each token the moment it exists.
The model generates tokens incrementally, and the gateway streams them straight to the browser over a persistent connection (server-sent events or WebSocket). The user sees the answer appear word by word. The metric that matters is time-to-first-token, not total time.
Step 8 · Keep it safe
Guard the input and the output
Users will send prompts that try to jailbreak, extract secrets, or produce harmful content — and even a well-behaved model can generate something unsafe. Where do you put the checks?
Where do safety checks belong in the request path?
Models can be jailbroken and make mistakes. Safety can’t be the model’s job alone — you need an independent gate.
Input filtering catches malicious prompts but not unsafe generations. A clean prompt can still yield output you must block. You need both sides.
A Guardrails layer checks the prompt before generation and the answer before it reaches the user — defense on both edges, independent of the model.
A Guardrails layer screens both edges: the incoming prompt (block jailbreaks, abuse, PII leaks) before it reaches the model, and the outgoing answer before it streams to the user. Safety flags are emitted to the event stream so abuse can be detected and rate-limited over time.
What the new pieces do
- Guardrailsservice
- Screens the incoming prompt and the outgoing answer for unsafe or disallowed content before either is acted on.
The payoff
You built a conversational AI
From one box to a production LLM system: a gateway that orchestrates, a context builder that budgets tokens, durable memory, a batched GPU serving tier, streaming, and guardrails on both edges.
Now overload the model servers and watch how the serving tier — isolated behind its own queue — sheds load and degrades to a smaller model instead of taking the whole product down.
Everything you assembled, in order
- Inference Gateway — one front door — auth, rate limits, orchestration
- Context Builder — assembles system prompt + history + message
- Token budget — pin, keep recent, summarize, retrieve (RAG)
- Memory Service + DB — history lives outside the stateless model
- Model Servers — batched GPU inference + KV cache
- Streaming — tokens pushed as generated — fast time-to-first-token
- Guardrails — screen both the prompt and the answer
- Event stream — usage + safety events → billing, evals, abuse detection