The big idea
What is a conversational AI, really?
Strip away the chat bubble and a system like ChatGPT does one deceptively simple thing: take your message plus everything said so far, run it through a language model, and stream back an answer — for millions of people at once.
That sentence hides the hard parts. What exactly do we feed the model (and how much fits)? How does it remember earlier turns? How do we serve a giant model fast and affordably? And how do we keep the output safe?
How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Then watch the diagram on the right grow. Hover any box, replay the flow, and at the end overload the model servers to see what breaks. Hit Begin.
Step 1 · The skeleton
A client, a gateway, a model
Your browser can’t hold a 100-billion-parameter model, and it can’t be trusted with the keys, the rate limits, or other users’ data. So where does the request go?
Design decision: You type a message. Where does the thinking happen?
The call: A gateway in the middle authenticates, then calls the model servers. — A neutral backend holds the keys, enforces limits, assembles the prompt and routes to the GPU fleet. This is the spine of every LLM product.
We put an Inference Gateway in the middle. The browser sends a turn; the gateway authenticates, rate-limits, and orchestrates everything behind it. This is the client → gateway → model backbone under every chat product.
Client–gateway–model: The browser is the client (it asks). The gateway is the orchestrator (auth, limits, routing). The model servers are the brain. Keep these roles separate and everything else gets easier.
Step 2 · What the model sees
Assemble the context window
A language model has no memory between calls — it only sees the text you hand it this request. So what exactly do we put in front of it for each turn?
Design decision: The model is stateless. What do you send it for each new message?
The call: A built prompt: system instructions + relevant history + the new message. — A Context Builder composes the model input each turn — the system prompt, the recent/relevant history, and the new message — shaped to fit the window.
A Context Builder composes the model input for every turn: the system prompt (who the assistant is), the conversation history, and the new message. The model is stateless, so the full context is rebuilt and resent on each request.
The context window: Everything the model "knows" in a turn is the text in its context window — a fixed maximum number of tokens. There is no hidden memory; if it isn’t in the window, the model can’t see it.
Step 3 · Budget the tokens
It won’t all fit — now what?
The window is finite (say 128k tokens) and every token costs money and latency. A long chat — or a big document — blows past it. How do you decide what makes the cut?
Design decision: The conversation is longer than the context window. What gets sent?
The call: Budget the window: keep the system prompt + recent turns, summarize or retrieve the rest. — Allocate the token budget deliberately — pin the system prompt, keep recent turns, compress older ones, and pull in only the relevant facts (RAG) instead of everything.
The builder works to a token budget. It pins the system prompt, keeps the most recent turns, and summarizes or drops older ones. For knowledge beyond the chat, it retrieves only the relevant passages from a vector store (RAG) instead of stuffing everything in.
Tokens are the currency: Cost and latency both scale with tokens in + tokens out. Treating the context window as a budget to allocate — not a bucket to fill — is the core discipline of LLM system design.
Step 4 · Make it remember
Where does history live?
The model is stateless and the browser can’t be trusted to hold the true history. Yet tomorrow the user reopens the chat and expects it all to be there. Where does the conversation actually live?
Design decision: The model forgets between calls. Where is the real conversation stored?
The call: A Memory Service backed by a durable Conversation DB. — Persist every turn server-side. The builder loads history from it each request and saves the new turn — survives reloads, devices and crashes.
A Memory Service reads and writes the Conversation DB — the durable source of truth for every turn. Each request, the builder loads recent history from it; after each answer, the new turn is appended. The browser is just a fast cache of what the database already knows.
Memory lives outside the model: Because inference never updates the weights, all "memory" is just text we store and re-feed. Persist it durably, reload it per turn — that’s what makes a stateless model feel like it remembers.
Step 5 · The real cost
Inference is the expensive part
Assembling a prompt is cheap. Running a hundred-billion-parameter model to generate each token is not — it needs GPUs, and a single request can hold one for seconds. How should the system treat this?
Design decision: What dominates the cost and latency of a chat turn?
The call: GPU inference — generating the answer token by token. — Each output token is a full forward pass through the model on scarce GPUs. This dominates both cost and latency, so it gets its own carefully-managed tier.
Generation runs on a dedicated Model Servers tier — a fleet of GPUs holding the model weights. Because each output token is a full pass through the model, this tier is the bottleneck we design everything else around: it gets its own queue, its own scaling, and its own failure handling.
Isolate the bottleneck: The most expensive, scarcest resource (GPU inference) becomes its own tier so it can be queued, batched, scaled and protected independently from the cheap stateless work around it.
Step 6 · Serve it efficiently
Batching and the KV cache
GPUs are most efficient when fully fed, but chat requests arrive one at a time and each generates tokens one at a time. Run them naively and the GPUs sit half-idle while users wait. How do you serve a giant model fast?
Design decision: How do you keep scarce GPUs busy across many concurrent chats?
The call: Continuously batch many requests together and reuse the KV cache. — Pack concurrent requests into shared batches and cache the attention state (KV cache) so each new token is cheap. This is how real serving stacks hit high throughput.
The model servers use continuous batching — packing many in-flight chats into shared GPU passes — and a KV cache that stores attention state so each new token reuses prior work instead of recomputing the whole prompt. Every turn also emits usage events (tokens, latency) to the stream for billing and evals.
Throughput vs latency: Batching raises throughput (chats/sec across the fleet) at a small cost to any single request’s latency. The KV cache makes per-token generation cheap. Together they make serving a frontier model economical.
Step 7 · Don’t make them wait
Stream the tokens back
A long answer can take many seconds to finish. If the user stares at a blank screen until it’s done, the product feels broken — even when it’s working perfectly. How do you make it feel instant?
Design decision: A full answer takes 6 seconds to generate. What does the user see?
The call: Tokens streamed to the screen as they’re generated. — Stream over a persistent connection (SSE/WebSocket) so words appear as they’re produced. Time-to-first-token is what users actually feel.
The model generates tokens incrementally, and the gateway streams them straight to the browser over a persistent connection (server-sent events or WebSocket). The user sees the answer appear word by word. The metric that matters is time-to-first-token, not total time.
Perceived latency: Streaming doesn’t make generation faster — it makes it feel instant by showing progress immediately. Optimizing the first token often beats optimizing the total.
Step 8 · Keep it safe
Guard the input and the output
Users will send prompts that try to jailbreak, extract secrets, or produce harmful content — and even a well-behaved model can generate something unsafe. Where do you put the checks?
Design decision: Where do safety checks belong in the request path?
The call: Screen both the incoming prompt and the outgoing answer. — A Guardrails layer checks the prompt before generation and the answer before it reaches the user — defense on both edges, independent of the model.
A Guardrails layer screens both edges: the incoming prompt (block jailbreaks, abuse, PII leaks) before it reaches the model, and the outgoing answer before it streams to the user. Safety flags are emitted to the event stream so abuse can be detected and rate-limited over time.
Defense at both edges: Treat safety as an independent layer around the model, not a property of the model. Check input and output — a clean prompt can still produce unsafe output, and vice versa.
The payoff
You built a conversational AI
From one box to a production LLM system: a gateway that orchestrates, a context builder that budgets tokens, durable memory, a batched GPU serving tier, streaming, and guardrails on both edges.
Now overload the model servers and watch how the serving tier — isolated behind its own queue — sheds load and degrades to a smaller model instead of taking the whole product down.
- Inference Gateway — one front door — auth, rate limits, orchestration
- Context Builder — assembles system prompt + history + message
- Token budget — pin, keep recent, summarize, retrieve (RAG)
- Memory Service + DB — history lives outside the stateless model
- Model Servers — batched GPU inference + KV cache
- Streaming — tokens pushed as generated — fast time-to-first-token
- Guardrails — screen both the prompt and the answer
- Event stream — usage + safety events → billing, evals, abuse detection