LABS · 20  /  SERVING

The Assembly Line.

A GPU serving one request is a factory running one item at a time — almost all of it idle. Batch the requests and throughput soars, because you stream the weights once for many sequences. But batches fight back with latency and memory — and naive batching stalls on its slowest member. Run the line and find the balance.

Act 1 · Why one request wastes a GPU
Stream the weights once, serve many
Generating a token means streaming every weight through the GPU — the compute barely registers next to that memory traffic. So a single request leaves the compute almost idle. Add more requests to the same pass and they ride the same weight-stream nearly for free. Slide the batch up.
Batch size (concurrent requests)1
— tok/s
Throughput
total tokens/sec
—%
GPU compute used
Cost / token
relative to batch 1
At batch 1 the GPU is a supercomputer doing a pocket-calculator's work. Each request you add rides the same weight-streaming pass, so throughput climbs and cost-per-token falls — the entire economic case for batching.
Act 2 · The trade
Throughput up, latency up too
Batching isn't free lunch forever. Bigger batches raise total throughput but each request waits to be gathered and shares the pass — so per-request latency climbs, and eventually the KV cache runs out of memory. Find the batch that hits your latency target.
Batch size16
Latency budget (per token)40 ms
Throughput
Per-token latency
Within budget?
Interactive chat wants small batches (low latency); offline bulk jobs want huge batches (max throughput). The serving system's job is to sit at the biggest batch that still clears your latency budget.
Act 3 · The straggler problem
Static batching stalls; continuous doesn't
Requests finish at different times. Static batching waits for the whole batch to complete — short requests finish and their slots sit idle behind the longest one. Continuous batching evicts a finished request instantly and admits a waiting one. Run both and watch the idle time.
Mode:
Generating Idle slot (wasted) Finished Waiting to admit
—%
Avg slot utilization
0
Requests completed
0
Steps elapsed
Static leaves a growing graveyard of idle slots as short requests finish and wait for the long one. Continuous refills instantly — several-fold more throughput from the same GPU. It's why vLLM and friends made it the default.

Why a busy GPU is a cheap GPU

The economics of LLM serving rest on one fact: decode is memory-bound. Producing a token streams every weight of the model through the processor, and that memory traffic — not arithmetic — dominates the time. A forward pass therefore has enormous spare compute: at batch size 1, the GPU's tensor cores are mostly idle while the weights crawl through.

Batching spends that spare compute. Run 32 requests in one forward pass and you stream the weights once but advance 32 sequences — roughly 32× the tokens for about the same memory cost. Throughput (tokens per second) is what sets your cost per token, so batching is the single biggest lever on serving economics. The limit is the KV cache: every concurrent request holds its own cache, so you can only batch as many as fit in VRAM.

Throughput ≠ latency

Two different clocks. Throughput (system tokens/sec) sets cost and rises with batch size. Latency (one request's tokens/sec) sets user experience and falls as batches grow, because each request shares the pass. You tune batch size to the biggest value that still meets a latency target — small for chat, huge for offline.

The straggler tax, and the fix

Requests have wildly different output lengths. Static batching runs a fixed group until all finish — so a batch with one 2,000-token request drags 31 short ones, whose freed slots sit idle to the end, and no new request can join. The GPU runs at the pace of its slowest member.

Continuous batching (a.k.a. in-flight batching) fixes it: the instant any sequence finishes, it's evicted and a waiting request takes its slot — every single step. The batch is never padded out for stragglers, utilization stays pinned high, and real-world throughput often jumps several-fold. It's the headline feature of modern serving stacks like vLLM-style engines, and the reason a well-run GPU serves so many more users than a naive one.

Check yourself

Four questions. Batch your best guesses.

Finished this one? 0 / 22 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs