Serving an LLM
fast, and cheap.

An LLM writes one token at a time, and to produce each token it must read every one of its billions of weights out of GPU memory. The arithmetic per token is trivial next to that data movement — so generation is memory-bandwidth-bound, not compute-bound. The GPU's math units sit mostly idle, waiting on memory.

This one fact explains almost every serving optimization. If the bottleneck is moving data, then the wins come from moving less of it (quantization), reusing what you've moved (the KV cache), and amortizing the move across many requests (batching). Keep "it's memory, not math" in mind and the whole field makes sense.

Every request has two phases with opposite characteristics, and serving systems treat them differently.

The split matters for two metrics you'll live by: time to first token (TTFT) is dominated by prefill, while tokens per second after that is a decode property. A long prompt slows your first token; a long answer slows everything after it.

Attention lets each new token look back at every previous token — specifically at their Key and Value vectors. Recomputing those for the whole history at every step would be madness, so they're computed once and kept around: the KV cache. It's what makes decode fast.

The catch is size. The cache grows with every token, every layer, and every request in the batch — and it lives in the same precious GPU memory as the model weights. On long contexts and big batches, the KV cache, not the model, becomes the thing that runs you out of memory. Managing it well is the central problem of modern LLM serving.

Because moving the weights is the cost, you want as many requests as possible sharing each trip — that's batching. But naive static batching waits for the whole batch to finish before starting the next, so one long request holds up a dozen short ones and the GPU idles.

Continuous batching (popularized by vLLM) fixes this: it adds and removes requests from the running batch at every token step. The instant one request finishes, a waiting one takes its slot — the GPU never stalls.

This is the single biggest throughput win in modern serving, and it's why open-source servers like vLLM became the default. It costs nothing in quality — the outputs are identical; you're just scheduling the GPU better.

Continuous batching creates a new problem: requests come and go, each with a KV cache that grows unpredictably. Allocating one big contiguous block per request wastes huge amounts of memory to fragmentation and over-provisioning.

PagedAttention borrows the operating-system idea of virtual memory. It chops the KV cache into small fixed-size blocks (pages) that don't have to be contiguous, and keeps a lookup table mapping each request's logical sequence to its scattered physical blocks. Memory is handed out a page at a time, exactly as needed — so almost none is wasted, and you can fit far more concurrent requests on the same GPU.

Since serving is memory-bound, storing the model in lower precision is a direct speedup: smaller weights mean less data to move per token, plus more room for the KV cache. Quantization converts 16-bit float weights down to 8-bit or even 4-bit integers.

Going to 4-bit can shrink a model to a quarter of its size, often letting it run on a single consumer GPU with only a modest quality dip. The art is choosing what to quantize and how — weights are easy, activations are trickier — to keep the loss below what your product can notice.

Decode is slow because it's sequential — one token, then the next. Speculative decoding breaks the sequence with a bet. A small, fast draft model guesses several tokens ahead; then the big model checks all of them in a single forward pass. Verifying in parallel is cheap, so:

The crucial property: the output is exactly what the big model would have produced alone — verification guarantees it. You get lower latency with zero quality change, as long as the draft model is right often enough to pay for itself.

A chat UI optimizes for TTFT and streams tokens so the wait feels short; a bulk document-processing job optimizes for raw throughput and happily runs huge batches. The last lever is routing: send easy requests to a small cheap model and only escalate hard ones to the big model — the job of an LLM gateway.