Continuous Batching
also: dynamic batching · in-flight batching
Serving many users per GPU by advancing all in-flight generations together, admitting new ones mid-flight.
Per-request LLM decoding wastes a GPU — it is memory-bound, not compute-bound. Continuous batching advances every in-flight request by one token per forward pass and slots new arrivals into freed positions immediately, no waiting for the batch to drain. It is the single biggest lever on tokens/sec per GPU, and therefore on cost per token.