Naive batching waits to collect a full batch of requests before running any of them — wasteful when requests generate wildly different numbers of tokens and finish at different times. Continuous batching instead slots a new request in the moment a GPU has a free spot (because an earlier request just finished generating), keeping GPU utilization high throughout — one of the biggest single throughput wins in modern LLM serving engines like vLLM.
Worked example: unlike static batching (wait for a fixed batch, run it to completion), continuous batching adds new requests and evicts finished ones every decode step, so the GPU never idles waiting for the slowest sequence. Gotcha: it is the key to high LLM serving throughput, but it makes per-request latency variable and scheduling complex — which is exactly the machinery engines like vLLM provide.