Batch vs real-time inference: throughput, or latency?

Batch inference runs the model over many inputs together, offline or asynchronously — a scheduled job or a queue drains a big pile of work and writes results somewhere. Nobody is waiting on any single prediction, so you optimize for throughput (predictions per second per GPU) and cost per token. Real-time inference (a.k.a. online) answers one request at a time, synchronously, while a user or a calling service waits — so you optimize for latency (time to first token, tail latency) even if it means the GPU sits underused between requests.

The old framing — "batch is efficient, online is slow" — has softened, because modern serving engines (vLLM, TGI and friends) use continuous batching: they dynamically pack many in-flight online requests into a shared batch on the GPU, adding and removing sequences as they start and finish. That recovers much of batch inference’s throughput while serving interactive traffic at low latency. So online serving is no longer synonymous with "one request, one underused GPU." Still, a dedicated offline batch job — no latency budget at all, maximal batch size — remains the cheapest way to push a fixed pile of work through a GPU.

Why is batch cheaper? GPUs are throughput machines: they hit peak efficiency when they have a large batch of work to chew through in parallel, and they waste cycles when fed one small request at a time. A naive online server that runs each request alone leaves the GPU mostly idle between and within requests — you pay for the whole card but use a fraction of it. Batch inference fills the GPU: big batches mean near-100% utilization, which is exactly why cost per token drops.

Continuous batching is the clever middle path — it keeps utilization high by always topping up the batch with whatever requests are in flight, so online serving stops leaving the GPU idle. But there’s still a floor: interactive traffic is bursty and latency-bounded, so you can’t always wait to assemble a perfectly full batch. A pure offline job has no such constraint, which is why "just run it as a batch overnight" remains the move whenever latency truly doesn’t matter.

The chat box itself is unavoidably real-time: a user typed a message and is watching for tokens, so you serve it from an online endpoint with continuous batching to keep latency low while still getting decent GPU utilization from concurrent users. Optimizing this for raw throughput at the expense of time-to-first-token would make the product feel broken, so latency wins here.

But the same product has a pile of non-interactive work: embedding the 10M documents behind its retrieval, re-scoring yesterday’s conversations for quality, generating weekly summary emails. None of that has a human waiting, so it runs as batch jobs — huge batches, maximal GPU utilization, lowest possible cost — scheduled off-peak. Same models, two serving modes, chosen per workload by the one question: is someone waiting?