The whole design, in writing
Learn AI system design by building an LLM inference serving system step by step. An interactive guide covering the request queue, the prefill/decode split, continuous batching, the KV cache and paged attention, tensor sharding across GPUs, autoscaling on GPU pressure, and the latency-versus-throughput trade-offs of serving a model at scale.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
Why is serving an LLM hard?
Running a model once on your laptop is easy. Serving it to thousands of users at once, fast and affordably, on hardware that costs dollars per hour, is not. A single request can hog a GPU for seconds while it generates tokens one at a time. How do you keep scarce GPUs full and users’ answers fast?
Treat the GPU fleet as a precious, fixed resource and design everything around it: a queue to absorb spikes, continuous batching to keep GPUs full, a KV cache to make each token cheap, sharding to fit big models, and autoscaling to ride demand.
What the new pieces do
- Clientclient
- Sends a prompt and reads tokens back as they stream. Cares about time-to-first-token and tokens/sec.
Step 1 · The skeleton
A client, a router, a model
A client sends a prompt and wants tokens streamed back. The model lives on GPUs that can’t be exposed directly. What sits in between?
Requests arrive for a model running on a GPU fleet. What fronts it?
No auth, no load balancing, no limits — and a client pinned to one worker that might be busy. You need a router in front.
The router authenticates, applies limits, and sends each request to a GPU replica that has room — then streams tokens back.
Generations are dynamic and rarely identical, so caching whole answers barely helps. The expensive part is fresh inference, not delivery.
A Router fronts the fleet: it admits requests, applies rate limits, and routes each to a GPU Worker with capacity, then streams the generated tokens back to the Client. The classic client → router → workers spine, tuned for streaming.
What the new pieces do
- Routerbackend
- The front door. Authenticates, applies limits, and routes each request to a model replica with capacity.
Step 2 · Absorb the spikes
Put a queue in front of the GPUs
Traffic is bursty — quiet, then a flood. GPUs are a fixed number. If requests hit workers directly, a burst either overwhelms them or, when quiet, leaves them idle. How do you smooth this?
GPU count is fixed; traffic is spiky. How do you avoid stampede AND starvation?
Dropping on every spike is a terrible experience and wastes capacity that frees up moments later. Buffer first, drop only as a last resort.
GPUs take minutes to provision and cost too much to hold per request. You can’t scale per-request at GPU granularity in real time.
A queue absorbs bursts so GPUs stay fully fed without being stampeded, and admission control can shed load gracefully when the queue gets too deep.
A Request Queue sits between the router and the GPUs. Bursts fill the queue instead of crushing the workers; in quiet moments the queue drains and GPUs stay busy. Admission control can shed load ("busy, retry") when the queue grows too deep — protecting latency for everyone already in flight.
What the new pieces do
- Request Queueservice
- Holds incoming requests so the GPUs are never starved or stampeded — the buffer that absorbs spiky traffic.
Step 3 · How a token is made
Prefill, then decode
To serve a model well you have to know how it actually computes. Generating an answer isn’t one operation — it’s two very different phases with very different costs. What are they?
Generating an answer splits into two phases. What are they?
Weights load once at startup, not per request. The per-request work is the two-phase prefill/decode, which dominates everything.
Prefill runs the full prompt through the model in parallel (compute-heavy); decode then generates output tokens one at a time (memory-bandwidth-heavy). They scale differently.
That’s a different architecture’s framing. Decoder-style LLM serving is specifically prefill (prompt) then autoregressive decode (output).
The GPU Workers do two phases. Prefill processes the entire prompt in one parallel pass (compute-bound). Decode then generates output one token per step, each step depending on the last (memory-bandwidth-bound). The asymmetry — fast parallel prefill, slow sequential decode — shapes every serving decision.
- 1 passprefill the prompt
- 1 tokenper decode step
- TTFTset by prefill
What the new pieces do
- GPU Workersservice
- The model running on GPUs. Does a one-time prefill of the prompt, then decodes output one token per step.
Step 4 · Keep the GPUs full
Continuous batching
Decode generates one token per step per sequence — a single request barely uses the GPU. But requests start and finish at different times and have different lengths. Naive batching wastes the GPU waiting for the slowest one. How do you keep it packed?
Requests have different lengths and arrive at different times. How do you batch?
A single decode step uses a fraction of the GPU; everything else waits. You’re paying for hardware that mostly idles.
The whole batch stalls on the longest sequence, and new arrivals wait for the next batch. Idle gaps everywhere.
A scheduler packs many sequences into each GPU step, dropping finished ones and slotting in new arrivals immediately. The GPU stays saturated regardless of length mix.
A Batch Scheduler does continuous (in-flight) batching: at each decode step it runs all active sequences together, retires the ones that just finished, and slots waiting requests in immediately — no waiting for a batch to drain. The GPU stays full no matter how lengths and arrivals mix. Serving metrics (TTFT, tokens/sec, utilization) drive its decisions.
What the new pieces do
- Batch Schedulerservice
- Packs many in-flight requests into each GPU step, adding and retiring sequences token by token.
- Metrics / Controlbus
- Streams serving metrics — time-to-first-token, throughput, GPU memory — that drive batching and autoscaling.
Step 5 · Make each token cheap
The KV cache and paged attention
At each decode step, attention needs to look back over every previous token. Recomputing that for the whole sequence on every single token would be brutally quadratic. And the cache that avoids it can blow up GPU memory. How do you make decode both fast and memory-efficient?
How do you avoid recomputing attention over the whole prompt every token?
That’s the quadratic waste the KV cache exists to kill — recomputing all prior tokens for every new one is enormously expensive.
Store the attention K/V per token so each new token reuses prior work — and page the cache in fixed blocks so many sequences pack into GPU memory without fragmentation.
Answers rarely repeat verbatim, so that barely helps. The win is caching intermediate attention state within a single generation.
The KV Cache stores each token’s attention key/value state so every new token reuses prior work instead of recomputing — turning decode from quadratic to linear. Paged attention stores that cache in fixed-size blocks (like OS virtual memory), so many sequences share GPU memory without waste — which is exactly what lets continuous batching pack so many requests in.
What the new pieces do
- KV Cachecache
- Per-sequence attention state kept in GPU memory so each new token reuses prior work instead of recomputing.
Back of the envelope
- KV cache
- reuse attention state — decode goes O(n), not O(n²)
- paged attention
- fixed-block cache → no fragmentation, dense packing
- memory caps concurrency
- more KV memory = bigger batches
- evict / preempt
- pause low-priority sequences when memory is tight
Step 6 · Fit a giant model
Shard the weights across GPUs
A frontier model’s weights don’t fit in a single GPU’s memory. You can’t shrink the model. So how do you run something bigger than any one device?
The model’s weights are larger than one GPU’s memory. Now what?
Sometimes valid — but when you need the big model, you must run it as-is. The serving system has to handle models bigger than one device.
Split each layer’s tensors across GPUs (tensor parallelism), with fast interconnect so they act as one logical worker. Pipeline parallelism splits by layer for even larger models.
Disk is orders of magnitude too slow for per-token weight access. Weights must live in GPU memory — sharded if they don’t fit on one.
The Model Weights are tensor-sharded across several GPUs: each layer’s matrices are split so the GPUs compute a forward pass together over fast interconnect, acting as one logical worker. Even larger models add pipeline parallelism (split by layer). The fleet is then many such multi-GPU workers behind the scheduler.
What the new pieces do
- Model Weightsstore
- The parameters, split across multiple GPUs when the model is too big for one device.
Step 7 · Ride the demand
Autoscale on GPU pressure
Demand swings through the day. Provision for the peak and you burn money on idle GPUs at 3am; provision for the average and you melt at peak. GPUs aren’t instant to add. How do you size the fleet?
Demand swings hour to hour and GPUs are slow to provision. How do you size the fleet?
You pay for peak capacity around the clock, idling expensive GPUs most of the day. Bleeds money.
The queue smooths small bursts, but a sustained peak just grows the queue and latency without end. Buffering isn’t capacity.
An autoscaler watches the real pressure signals and adds/removes GPU replicas, keeping a warm pool so scale-up isn’t cold-start slow. Match capacity to demand.
An Autoscaler watches the true pressure signals — queue depth and GPU utilization — and adds or removes replicas to hold latency targets. Because GPUs are slow to start, it keeps a warm pool and scales ahead of the curve. Under extreme load it sheds or routes overflow to a smaller, faster model.
What the new pieces do
- Autoscalerservice
- Watches queue depth and GPU utilization and adds or removes replicas to hold latency under load.
The payoff
You built an inference server
From "run it once" to serving at scale: a router and queue, the prefill/decode split, continuous batching, a paged KV cache, tensor-sharded weights, and autoscaling — all balancing the latency/throughput trade-off.
Now flood the queue and feel the central tension: as load climbs, time-to-first-token rises, and you watch batching, the KV cache, and autoscaling fight to protect tail latency before the fleet stalls.
Everything you assembled, in order
- Router + stream — admit, route, stream tokens — optimize TTFT
- Request Queue — shock absorber between spiky demand and fixed GPUs
- Prefill / decode — parallel prompt pass, then one token per step
- Continuous batching — add/retire sequences each step — GPUs stay full
- KV cache + paging — cheap tokens, dense concurrency, memory-bound
- Tensor sharding — split big models across GPUs to fit and scale
- Autoscaler — scale on queue depth + GPU util, keep warm pools
- Latency vs throughput — the trade-off every knob above is balancing
