The big idea
Why is serving an LLM hard?
Running a model once on your laptop is easy. Serving it to thousands of users at once, fast and affordably, on hardware that costs dollars per hour, is not. A single request can hog a GPU for seconds while it generates tokens one at a time. How do you keep scarce GPUs full and users’ answers fast?
Treat the GPU fleet as a precious, fixed resource and design everything around it: a queue to absorb spikes, continuous batching to keep GPUs full, a KV cache to make each token cheap, sharding to fit big models, and autoscaling to ride demand.
How to read this: Each step opens with a real design decision — you make the call before I show you what ships. Watch the diagram grow, hover any box, replay the flow. At the end flood the queue to feel the latency/throughput tension. Hit Begin.
Step 1 · The skeleton
A client, a router, a model
A client sends a prompt and wants tokens streamed back. The model lives on GPUs that can’t be exposed directly. What sits in between?
Design decision: Requests arrive for a model running on a GPU fleet. What fronts it?
The call: A router that admits requests and routes them to a replica with capacity. — The router authenticates, applies limits, and sends each request to a GPU replica that has room — then streams tokens back.
A Router fronts the fleet: it admits requests, applies rate limits, and routes each to a GPU Worker with capacity, then streams the generated tokens back to the Client. The classic client → router → workers spine, tuned for streaming.
Stream-first serving: Inference responses arrive token by token over seconds, so the path is built around streaming. The metric that matters most isn’t total time — it’s time-to-first-token.
Step 2 · Absorb the spikes
Put a queue in front of the GPUs
Traffic is bursty — quiet, then a flood. GPUs are a fixed number. If requests hit workers directly, a burst either overwhelms them or, when quiet, leaves them idle. How do you smooth this?
Design decision: GPU count is fixed; traffic is spiky. How do you avoid stampede AND starvation?
The call: Buffer requests in a queue the scheduler pulls from. — A queue absorbs bursts so GPUs stay fully fed without being stampeded, and admission control can shed load gracefully when the queue gets too deep.
A Request Queue sits between the router and the GPUs. Bursts fill the queue instead of crushing the workers; in quiet moments the queue drains and GPUs stay busy. Admission control can shed load ("busy, retry") when the queue grows too deep — protecting latency for everyone already in flight.
The queue is the shock absorber: A fixed GPU fleet can’t flex instantly, so the queue decouples arrival rate from service rate. It’s what lets spiky demand meet steady hardware without stampede or starvation.
Step 3 · How a token is made
Prefill, then decode
To serve a model well you have to know how it actually computes. Generating an answer isn’t one operation — it’s two very different phases with very different costs. What are they?
Design decision: Generating an answer splits into two phases. What are they?
The call: Prefill (process the whole prompt at once) then decode (one token per step). — Prefill runs the full prompt through the model in parallel (compute-heavy); decode then generates output tokens one at a time (memory-bandwidth-heavy). They scale differently.
The GPU Workers do two phases. Prefill processes the entire prompt in one parallel pass (compute-bound). Decode then generates output one token per step, each step depending on the last (memory-bandwidth-bound). The asymmetry — fast parallel prefill, slow sequential decode — shapes every serving decision.
Two phases, two bottlenecks: Prefill is compute-bound and bursty; decode is bandwidth-bound and long. Time-to-first-token is mostly prefill; tokens/sec is decode. You optimize them separately.
Step 4 · Keep the GPUs full
Continuous batching
Decode generates one token per step per sequence — a single request barely uses the GPU. But requests start and finish at different times and have different lengths. Naive batching wastes the GPU waiting for the slowest one. How do you keep it packed?
Design decision: Requests have different lengths and arrive at different times. How do you batch?
The call: Continuous batching — add and retire sequences every step. — A scheduler packs many sequences into each GPU step, dropping finished ones and slotting in new arrivals immediately. The GPU stays saturated regardless of length mix.
A Batch Scheduler does continuous (in-flight) batching: at each decode step it runs all active sequences together, retires the ones that just finished, and slots waiting requests in immediately — no waiting for a batch to drain. The GPU stays full no matter how lengths and arrivals mix. Serving metrics (TTFT, tokens/sec, utilization) drive its decisions.
Throughput vs latency: Bigger batches raise throughput (tokens/sec across the fleet) but can nudge any one request’s latency up. Continuous batching captures most of the throughput win with minimal latency cost — the central trade-off of serving.
Step 5 · Make each token cheap
The KV cache and paged attention
At each decode step, attention needs to look back over every previous token. Recomputing that for the whole sequence on every single token would be brutally quadratic. And the cache that avoids it can blow up GPU memory. How do you make decode both fast and memory-efficient?
Design decision: How do you avoid recomputing attention over the whole prompt every token?
The call: Cache each token’s key/value state and page it like virtual memory. — Store the attention K/V per token so each new token reuses prior work — and page the cache in fixed blocks so many sequences pack into GPU memory without fragmentation.
The KV Cache stores each token’s attention key/value state so every new token reuses prior work instead of recomputing — turning decode from quadratic to linear. Paged attention stores that cache in fixed-size blocks (like OS virtual memory), so many sequences share GPU memory without waste — which is exactly what lets continuous batching pack so many requests in.
Memory is the real limit: Decode is bound by GPU memory bandwidth and capacity, not raw compute. The KV cache makes each token cheap; paging it makes concurrency dense. How many requests you can batch is mostly a KV-cache-memory question.
Step 6 · Fit a giant model
Shard the weights across GPUs
A frontier model’s weights don’t fit in a single GPU’s memory. You can’t shrink the model. So how do you run something bigger than any one device?
Design decision: The model’s weights are larger than one GPU’s memory. Now what?
The call: Tensor-shard the weights across multiple GPUs that compute together. — Split each layer’s tensors across GPUs (tensor parallelism), with fast interconnect so they act as one logical worker. Pipeline parallelism splits by layer for even larger models.
The Model Weights are tensor-sharded across several GPUs: each layer’s matrices are split so the GPUs compute a forward pass together over fast interconnect, acting as one logical worker. Even larger models add pipeline parallelism (split by layer). The fleet is then many such multi-GPU workers behind the scheduler.
Parallelism, two ways: Tensor parallelism splits each layer within a step (needs fast interconnect); pipeline parallelism splits the model across layers/stages. Big-model serving combines them to fit and to scale.
Step 7 · Ride the demand
Autoscale on GPU pressure
Demand swings through the day. Provision for the peak and you burn money on idle GPUs at 3am; provision for the average and you melt at peak. GPUs aren’t instant to add. How do you size the fleet?
Design decision: Demand swings hour to hour and GPUs are slow to provision. How do you size the fleet?
The call: Autoscale replicas on queue depth and GPU utilization, with warm pools. — An autoscaler watches the real pressure signals and adds/removes GPU replicas, keeping a warm pool so scale-up isn’t cold-start slow. Match capacity to demand.
An Autoscaler watches the true pressure signals — queue depth and GPU utilization — and adds or removes replicas to hold latency targets. Because GPUs are slow to start, it keeps a warm pool and scales ahead of the curve. Under extreme load it sheds or routes overflow to a smaller, faster model.
Scale on the right signal: CPU-style metrics don’t capture GPU saturation. Autoscale on queue depth and GPU/KV-cache utilization — the signals that actually predict rising latency — and keep warm capacity because cold GPU starts are slow.
The payoff
You built an inference server
From "run it once" to serving at scale: a router and queue, the prefill/decode split, continuous batching, a paged KV cache, tensor-sharded weights, and autoscaling — all balancing the latency/throughput trade-off.
Now flood the queue and feel the central tension: as load climbs, time-to-first-token rises, and you watch batching, the KV cache, and autoscaling fight to protect tail latency before the fleet stalls.
- Router + stream — admit, route, stream tokens — optimize TTFT
- Request Queue — shock absorber between spiky demand and fixed GPUs
- Prefill / decode — parallel prompt pass, then one token per step
- Continuous batching — add/retire sequences each step — GPUs stay full
- KV cache + paging — cheap tokens, dense concurrency, memory-bound
- Tensor sharding — split big models across GPUs to fit and scale
- Autoscaler — scale on queue depth + GPU util, keep warm pools
- Latency vs throughput — the trade-off every knob above is balancing