Continuous Batching
also: in-flight batching
Adding and removing requests from a running batch on the fly, instead of waiting for a fixed batch to fill before starting.
Naive batching waits to collect a full batch of requests before running any of them — wasteful when requests generate wildly different numbers of tokens and finish at different times. Continuous batching instead slots a new request in the moment a GPU has a free spot (because an earlier request just finished generating), keeping GPU utilization high throughout — one of the biggest single throughput wins in modern LLM serving engines like vLLM.