The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a conversational AI, really?
Strip away the chat bubble and a system like ChatGPT does one deceptively simple thing: take your message plus everything said so far, run it through a language model, and stream back an answer — for millions of people at once.
That sentence hides the hard parts. What exactly do we feed the model (and how much fits)? How does it remember earlier turns? How do we serve a giant model fast and affordably? And how do we keep the output safe?
What the new pieces do
- Userclient
- A person typing a message. Sends a turn, then watches the answer stream back token by token.
Step 1 · The skeleton
A client, a gateway, a model
Your browser can’t hold a 100-billion-parameter model, and it can’t be trusted with the keys, the rate limits, or other users’ data. So where does the request go?
You type a message. Where does the thinking happen?
Frontier models are tens to hundreds of gigabytes and need GPUs. Even if it fit, you’d ship the weights to every user and lose all control over cost and safety.
A neutral backend holds the keys, enforces limits, assembles the prompt and routes to the GPU fleet. This is the spine of every LLM product.
Letting clients hit inference directly means no auth boundary, no rate limiting, no prompt assembly, and your model endpoint exposed to the world.
We put an Inference Gateway in the middle. The browser sends a turn; the gateway authenticates, rate-limits, and orchestrates everything behind it. This is the client → gateway → model backbone under every chat product.
Why this piece earns its place
The gateway earns its keep the moment you try to limit anything. Rate limiting by requests is the reflex, and here it is close to meaningless: a limiter that counts requests treats a one-line question and a turn carrying a pasted-in document as the same unit of work, and they are nowhere near the same cost. The limiter has to count tokens, which it can only do once the prompt is assembled and the answer is finished. So the gateway reserves a pessimistic estimate at the door and settles the real number when the turn completes, which is a different shape of limiter from the one every other service on your estate runs. The second thing it owns is the request contract: which knobs a caller may set, and which the server pins. Let a client choose a model or ask for a shorter answer, but keep the ceiling on output length on the server, because that one field multiplies what a turn costs and a caller has no reason to be modest with it. Defaults belong here too. A default that lives in client code is a default you cannot change — old clients keep sending the old value long after you changed your mind.
What the new pieces do
- Inference Gatewaybackend
- The single front door for every chat request. Authenticates, rate-limits, then orchestrates context, guardrails and model serving.
Step 2 · What the model sees
Assemble the context window
A language model has no memory between calls — it only sees the text you hand it this request. So what exactly do we put in front of it for each turn?
The model is stateless. What do you send it for each new message?
Then it forgets the system instructions and everything said earlier — every turn starts from scratch. The "conversation" disappears.
Right instinct, wrong limit. Conversations grow past the context window and cost scales with every token sent. You need to assemble and budget.
A Context Builder composes the model input each turn — the system prompt, the recent/relevant history, and the new message — shaped to fit the window.
A Context Builder composes the model input for every turn: the system prompt (who the assistant is), the conversation history, and the new message. The model is stateless, so the full context is rebuilt and resent on each request.
Why this piece earns its place
Treat the assembled prompt as a build artifact, not a string. The system prompt is the clearest case: because the builder reassembles everything on each turn, editing it silently changes the behavior of every conversation already in flight, including ones a user started yesterday and will carry on tomorrow. There is no migration, no marker anyone can see, and nothing in a log that says why quality moved. Version the system prompt and stamp that version onto the turn, and you can at least tell a prompt change apart from a model change when the numbers shift. The other thing that belongs here is that token counting is model-specific. Each model tokenizes with its own vocabulary, so the same history measures differently depending on which model the turn is routed to, and a budget computed against one model does not automatically hold for another. Count with the tokenizer of the model you are actually about to call, which means the target model is an input to the builder, not something it assumes.
What the new pieces do
- Context Builderservice
- Assembles the model input: system prompt + recent history + the new message, trimmed to fit the context window.
Step 3 · Budget the tokens
It won’t all fit — now what?
The window is finite (say 128k tokens) and every token costs money and latency. A long chat — or a big document — blows past it. How do you decide what makes the cut?
The conversation is longer than the context window. What gets sent?
Overflowing the window errors or silently drops the start — usually the system prompt, the most important part. You must decide what to keep on purpose.
Allocate the token budget deliberately — pin the system prompt, keep recent turns, compress older ones, and pull in only the relevant facts (RAG) instead of everything.
Recency usually matters most in a conversation, and the oldest turns are often stale. Blindly keeping the start wastes budget on what no longer matters.
The builder works to a token budget. It pins the system prompt, keeps the most recent turns, and summarizes or drops older ones. For knowledge beyond the chat, it retrieves only the relevant passages from a vector store (RAG) instead of stuffing everything in.
Why this piece earns its place
Two things about the budget are far easier to get right now than to retrofit. The first is order. Whatever is constant across turns belongs at the front and whatever changes every turn belongs at the back, because the serving tier keeps the prefixes it has already computed and reuses one whenever the leading tokens of a new request match it — across concurrent users, not merely against the request it happened to handle last. One shared system prompt at the front is therefore paid for once and reused by everybody chatting at the same time. Put the current date or a freshly retrieved passage in the second line and the match ends there, so you have invalidated everything after it, on every turn, for every user. Prompt layout looks like a formatting decision and is really a serving decision. The second is that summarizing older turns is itself a model call, with its own cost, latency and failure mode. Run it inline and you have put a generation in front of a turn someone is waiting on; run it in the background once a turn completes and the cost lands where no cursor is blinking. Keep the raw turns either way. The summary is derived and can be rebuilt with better rules later, and the day it becomes authoritative is the day you quietly discard detail no future turn can recover.
- 128kwindow (tokens)
- ~4 chars≈ 1 token
- in + outboth billed
What the new pieces do
- Vector / KBindex
- Optional knowledge source. The context builder can retrieve relevant facts to ground the answer (RAG).
Back of the envelope
- system prompt
- pinned — always sent
- recent turns
- kept verbatim while they fit
- older turns
- summarized into a running recap
- external knowledge
- retrieved on demand (RAG), not preloaded
Step 4 · Make it remember
Where does history live?
The model is stateless and the browser can’t be trusted to hold the true history. Yet tomorrow the user reopens the chat and expects it all to be there. Where does the conversation actually live?
The model forgets between calls. Where is the real conversation stored?
Inference doesn’t update the weights. The model never remembers anything between requests; memory must live outside it.
Close the tab and it’s gone; switch devices and it’s gone; you can’t rebuild context server-side. The client is a cache, not the source of truth.
Persist every turn server-side. The builder loads history from it each request and saves the new turn — survives reloads, devices and crashes.
A Memory Service reads and writes the Conversation DB — the durable source of truth for every turn. Each request, the builder loads recent history from it; after each answer, the new turn is appended. The browser is just a fast cache of what the database already knows.
Why this piece earns its place
The interesting part of memory is the write, not the read. Append the user message before generation and a crash leaves a turn with a question and no answer; append after and a crash loses the message the user actually typed. Write first and mark the assistant turn incomplete until generation finishes, then decide deliberately what a reopened conversation shows and whether a half-generated answer is fed back as context next turn. Nobody specifies this, and every chat product ends up answering it by accident. Retries need the same care. A client resending after a timeout appends a duplicate unless the turn carries an id the client generates, so the write is idempotent on that id rather than on arrival order. And history stops being one table the moment the product succeeds: the same text now sits in the database, inside the running summary, and in whatever prompt logging you added to debug bad answers. A deletion that only touches the first of those is not a deletion.
What the new pieces do
- Memory Serviceservice
- Loads and saves conversation history so a chat remembers earlier turns across requests.
- Conversation DBstore
- Durable storage of every message in every conversation — the source of truth for history.
Step 5 · The real cost
Inference is the expensive part
Assembling a prompt is cheap. Running a hundred-billion-parameter model to generate each token is not — it needs GPUs, and a single request can hold one for seconds. How should the system treat this?
What dominates the cost and latency of a chat turn?
A few small reads are microseconds. Real, but a rounding error next to generation.
Each output token is a full forward pass through the model on scarce GPUs. This dominates both cost and latency, so it gets its own carefully-managed tier.
Bytes are tiny; latency here is milliseconds. The seconds you wait are the model thinking, not the wire.
Generation runs on a dedicated Model Servers tier — a fleet of GPUs holding the model weights. Because each output token is a full pass through the model, this tier is the bottleneck we design everything else around: it gets its own queue, its own scaling, and its own failure handling.
Why this piece earns its place
What separates this tier from the stateless services around it is how slowly it grows. Adding a replica means pulling tens of gigabytes of weights and loading them into GPU memory, so new capacity is minutes away, not seconds. Autoscaling on queue depth therefore reacts well after the queue that triggered it has already hurt somebody — and the reaction can fail outright, because scarce accelerators are a reservation with a lead time, not an API call that always succeeds. Capacity here is a forecast you commit to ahead of the traffic, closer to planning a purchase than to tuning a control loop. The economics invert too. Over-provisioning a stateless tier is cheap enough to be standard advice; over-provisioning GPUs is the largest line on the bill, and an idle GPU costs exactly what a busy one does. So the number you manage is utilization, and that is a scheduling problem rather than a scaling one. Isolating serving also buys optionality you will want later: once it is its own tier behind the gateway, several pools — different model sizes, different versions, a canary — sit behind one front door, and no caller has to know which one answered.
Step 6 · Serve it efficiently
Batching and the KV cache
GPUs are most efficient when fully fed, but chat requests arrive one at a time and each generates tokens one at a time. Run them naively and the GPUs sit half-idle while users wait. How do you serve a giant model fast?
How do you keep scarce GPUs busy across many concurrent chats?
A single chat can’t saturate a GPU, and others queue behind it. You pay for hardware that mostly idles while users wait.
Pack concurrent requests into shared batches and cache the attention state (KV cache) so each new token is cheap. This is how real serving stacks hit high throughput.
Re-running attention over the entire context for each output token is quadratic waste — exactly what the KV cache exists to avoid.
The model servers use continuous batching — packing many in-flight chats into shared GPU passes — and a KV cache that stores attention state so each new token reuses prior work instead of recomputing the whole prompt. Every turn also emits usage events (tokens, latency) to the stream for billing and evals.
Why this piece earns its place
The ceiling on batch size is memory, not compute. Every sequence in the batch holds its own KV cache, and that cache grows with each token generated, so a replica’s real capacity is how many concurrent contexts of this length fit in GPU memory — and it shrinks as the conversations in flight get longer. A few very long chats can squeeze everyone else out of the batch, which surfaces as latency variance rather than as an error, spread across users who did nothing unusual. Two consequences are worth saying out loud. First, a request’s latency depends on what else is batched with it, so a latency target only means something at a stated concurrency, and a p99 measured on an idle fleet is a number about nothing. Second, when memory runs short a serving stack preempts sequences and recomputes their cache later, so a request can pay for its prompt twice with no failure recorded anywhere. It also turns the context budget from Step 3 into a capacity decision. Long contexts do not merely cost more per turn; they cost the fleet concurrency.
What the new pieces do
- Model Serversservice
- The GPU fleet running the LLM. Batches concurrent requests and reuses the KV cache to generate tokens efficiently.
- Model Weightsstore
- The model parameters, loaded across the GPU fleet. Large models are sharded over many GPUs.
- Event Streambus
- Every turn emits events — tokens used, latency, safety flags — feeding billing, evals and abuse detection.
Back of the envelope
- continuous batching
- merge concurrent requests into shared GPU passes
- KV cache
- reuse attention state — no recompute per token
- weights sharded across GPUs
- big models span many devices
- emit usage events
- tokens + latency → billing, evals, abuse detection
Step 7 · Don’t make them wait
Stream the tokens back
A long answer can take many seconds to finish. If the user stares at a blank screen until it’s done, the product feels broken — even when it’s working perfectly. How do you make it feel instant?
A full answer takes 6 seconds to generate. What does the user see?
Six seconds of blank screen reads as "frozen." You’re hiding progress the model is already making.
Stream over a persistent connection (SSE/WebSocket) so words appear as they’re produced. Time-to-first-token is what users actually feel.
Polling adds latency and load and still lags the model. A push stream delivers each token the moment it exists.
The model generates tokens incrementally, and the gateway streams them straight to the browser over a persistent connection (server-sent events or WebSocket). The user sees the answer appear word by word. The metric that matters is time-to-first-token, not total time.
Why this piece earns its place
Streaming moves most error handling out of HTTP. The status code is committed the instant the first token leaves, so everything that goes wrong afterwards — a model server dying, a guardrail firing, a timeout — has to be expressed inside the stream and handled by a client that is already showing text. There is no transparent retry once bytes are on the wire. You either resume from where it stopped or you visibly start over, and which one you pick is more a product decision than an engineering one. For the same reason the stream needs an explicit end-of-answer event: with a 200 long since sent, a connection that simply closes looks exactly like an answer that finished. Cancellation is the other half of it. Close the tab and nothing stops generation unless the gateway notices the dropped connection and propagates a cancel to the model server — and whether it can notice at all depends on what sits in front of it, since any proxy or CDN that buffers responses will hold that connection open, collect the whole answer, and hand it over in one lump long after the browser is gone. Tokens generated for a reader who left are the only work in this system with nobody at the end of it, and somebody still has to decide who pays for them.
Step 8 · Keep it safe
Guard the input and the output
Users will send prompts that try to jailbreak, extract secrets, or produce harmful content — and even a well-behaved model can generate something unsafe. Where do you put the checks?
Where do safety checks belong in the request path?
Models can be jailbroken and make mistakes. Safety can’t be the model’s job alone — you need an independent gate.
Input filtering catches malicious prompts but not unsafe generations. A clean prompt can still yield output you must block. You need both sides.
A Guardrails layer checks the prompt before generation and the answer before it reaches the user — defense on both edges, independent of the model.
A Guardrails layer screens both edges: the incoming prompt (block jailbreaks, abuse, PII leaks) before it reaches the model, and the outgoing answer before it streams to the user. Safety flags are emitted to the event stream so abuse can be detected and rate-limited over time.
Why this piece earns its place
Output screening and streaming pull against each other, and this is where the design is actually decided. Checking an answer the user has already read is not a check, so you either hold a chunk at a time before releasing it — spending the exact metric the previous step optimized — or you screen as you go and accept that you will occasionally retract text already on screen. Choose one on purpose; the default is to stumble into the second. It is also worth naming that a guardrail is a model. It has its own latency, it sits on the critical path twice, and it has its own failure modes, so it needs its own eval set and its own thresholds rather than being tuned by eye. The failure nobody sees is the false positive. A wrongly blocked answer looks identical in your metrics to a correctly blocked one, and the refused user just leaves. Reviewing those needs the flagged text, which is precisely the content you are least free to retain — so sample it, keep categories and scores in the clear, and put the text itself behind tighter access and a shorter window.
What the new pieces do
- Guardrailsservice
- Screens the incoming prompt and the outgoing answer for unsafe or disallowed content before either is acted on.
The payoff
You built a conversational AI
From one box to a production LLM system: a gateway that orchestrates, a context builder that budgets tokens, durable memory, a batched GPU serving tier, streaming, and guardrails on both edges.
Now overload the model servers and watch how the serving tier — isolated behind its own queue — sheds load and degrades to a smaller model instead of taking the whole product down.
Everything you assembled, in order
- Inference Gateway — one front door — auth, rate limits, orchestration
- Context Builder — assembles system prompt + history + message
- Token budget — pin, keep recent, summarize, retrieve (RAG)
- Memory Service + DB — history lives outside the stateless model
- Model Servers — batched GPU inference + KV cache
- Streaming — tokens pushed as generated — fast time-to-first-token
- Guardrails — screen both the prompt and the answer
- Event stream — usage + safety events → billing, evals, abuse detection
