Per-request LLM decoding wastes a GPU — it is memory-bound, not compute-bound. Continuous batching advances every in-flight request by one token per forward pass and slots new arrivals into freed positions immediately, no waiting for the batch to drain. It is the single biggest lever on tokens/sec per GPU, and therefore on cost per token.
Worked example: instead of one request per forward pass, a server batches many together so the GPU processes them in parallel — since decode is memory-bound, batching raises throughput dramatically at a small latency cost. Gotcha: naive static batching stalls fast requests behind slow ones; continuous batching (adding and removing sequences each step) is what modern servers use to keep the batch full without that head-of-line blocking.