A GPU serving one request is a factory running one item at a time — almost all of it idle. Batch the requests and throughput soars, because you stream the weights once for many sequences. But batches fight back with latency and memory — and naive batching stalls on its slowest member. Run the line and find the balance.
Act 1 · Why one request wastes a GPU
Stream the weights once, serve many
Generating a token means streaming every weight through the GPU — the compute barely registers next to that memory traffic. So a single request leaves the compute almost idle. Add more requests to the same pass and they ride the same weight-stream nearly for free. Slide the batch up.
At batch 1 the GPU is a supercomputer doing a pocket-calculator's work. Each request you add rides the same weight-streaming pass, so throughput climbs and cost-per-token falls — the entire economic case for batching.
Act 2 · The trade
Throughput up, latency up too
Batching isn't free lunch forever. Bigger batches raise total throughput but each request waits to be gathered and shares the pass — so per-request latency climbs, and eventually the KV cache runs out of memory. Find the batch that hits your latency target.
Batch size16
Latency budget (per token)40 ms
—
Throughput
—
Per-token latency
—
Within budget?
Interactive chat wants small batches (low latency); offline bulk jobs want huge batches (max throughput). The serving system's job is to sit at the biggest batch that still clears your latency budget.
Act 3 · The straggler problem
Static batching stalls; continuous doesn't
Requests finish at different times. Static batching waits for the whole batch to complete — short requests finish and their slots sit idle behind the longest one. Continuous batching evicts a finished request instantly and admits a waiting one. Run both and watch the idle time.
Mode:
Generating Idle slot (wasted) Finished Waiting to admit
—%
Avg slot utilization
0
Requests completed
0
Steps elapsed
Static leaves a growing graveyard of idle slots as short requests finish and wait for the long one. Continuous refills instantly — several-fold more throughput from the same GPU. It's why vLLM and friends made it the default.
Why a busy GPU is a cheap GPU
The economics of LLM serving rest on one fact: decode is memory-bound. Producing a token streams every weight of the model through the processor, and that memory traffic — not arithmetic — dominates the time. A forward pass therefore has enormous spare compute: at batch size 1, the GPU's tensor cores are mostly idle while the weights crawl through.
Batching spends that spare compute. Run 32 requests in one forward pass and you stream the weights once but advance 32 sequences — roughly 32× the tokens for about the same memory cost. Throughput (tokens per second) is what sets your cost per token, so batching is the single biggest lever on serving economics. The limit is the KV cache: every concurrent request holds its own cache, so you can only batch as many as fit in VRAM.
Throughput ≠ latency
Two different clocks. Throughput (system tokens/sec) sets cost and rises with batch size. Latency (one request's tokens/sec) sets user experience and falls as batches grow, because each request shares the pass. You tune batch size to the biggest value that still meets a latency target — small for chat, huge for offline.
The straggler tax, and the fix
Requests have wildly different output lengths. Static batching runs a fixed group until all finish — so a batch with one 2,000-token request drags 31 short ones, whose freed slots sit idle to the end, and no new request can join. The GPU runs at the pace of its slowest member.
Continuous batching (a.k.a. in-flight batching) fixes it: the instant any sequence finishes, it's evicted and a waiting request takes its slot — every single step. The batch is never padded out for stragglers, utilization stays pinned high, and real-world throughput often jumps several-fold. It's the headline feature of modern serving stacks like vLLM-style engines, and the reason a well-run GPU serves so many more users than a naive one.
Continuous batching, step by step
What the scheduler in a serving engine does on every iteration, the arithmetic that sets a decode step's speed, and a static-versus-continuous run worked by hand.
One scheduler iteration
Admit. While a slot and enough KV-cache memory are free, take the next waiting request and run its prompt through the model (prefill), in chunks if the prompt is long.
Step. Run one forward pass over every running sequence; each one gains exactly one token.
Retire. Any sequence that emitted its end token or hit its length cap leaves now, freeing its slot and its cache.
Protect memory. If the cache would overflow, preempt the newest sequence and requeue it rather than crash the batch.
Static batching skips steps 1 and 3 until the whole batch is done, so its slot utilization is Σ Lᵢ / (B · max Lᵢ) for output lengths Lᵢ and batch size B. In the memory-bound regime a decode step can be no faster than (weight bytes + KV bytes read) / bandwidth; throughput is B divided by that time, and each request's own speed is one token per step.
Worked example
Static. Four slots, eight requests with output lengths 20, 50, 100, 400, 20, 50, 100, 400. The first batch holds all four slots for 400 steps but fills only 570 of its 1,600 slot-steps, 35.6%. The second batch runs from step 400 to 800. Mean completion: 342.5 steps.
Continuous. The 20-token request frees its slot at step 20 and the next request starts there at once. Completions come at steps 20, 40, 50, 90, 100, 150, 400 and 490: a mean of 167.5 steps, half the latency, with everything done in 490 steps instead of 800.
A real step. Llama 3 8B in bf16 is 16.06 GB of weights. At 3.35 TB/s (an H100 SXM) one step takes at least 4.9 ms at batch 1 with a 2,000-token context: about 205 tokens per second.
Batch 32 at 2,000 tokens each adds 8.39 GB of cache reads (128 KiB per token): at least 7.3 ms per step, 4,385 tokens per second in total but 137 per request.
Long contexts erode the deal: at 8,000 tokens the same batch reads 33.6 GB of cache per step, twice the weights, and drops to 2,161 tokens per second.
The widget's Act 2 is a teaching model with the same shape: at its defaults, batch 16 gives 660 tokens per second at 32 ms per token, and the largest batch inside the 40 ms budget is 25, at 904 tokens per second.
Knobs and pitfalls
Engines expose a cap on concurrent sequences and on tokens processed per step (in vLLM, max_num_seqs and max_num_batched_tokens). A long prompt's prefill can stall every running decode, so chunked prefill splits it across iterations. Report time to first token and time per output token at p50 and p99, not throughput alone, and watch for preemption churn when caches run full. For offline jobs, push the batch as high as memory allows.
Check yourself
Four questions. Batch your best guesses.
See every answer, with the reasoning
Why does batching raise a GPU’s throughput so much?
Answer: Decode is memory-bound, so many requests ride one weight-streaming pass nearly for free — Stream the weights once, advance many sequences — the spare compute was there all along.
What happens to per-request latency as batch size grows?
Answer: It rises — each request waits to be gathered and shares the pass — Throughput and latency move opposite ways; you tune batch to a latency target.
Why is static batching wasteful?
Answer: Short requests finish and their slots idle until the batch’s slowest request completes — The batch runs at the pace of its slowest member; freed slots sit idle.
What does continuous (in-flight) batching do?
Answer: Evicts a finished request instantly and admits a waiting one, keeping the GPU full — No padding for stragglers → high utilization → several-fold more throughput. vLLM’s default.