The big idea

What is batch inference?

Sometimes you don't need an answer now — you need millions of answers, cheaply, by tomorrow: classify every support ticket, summarize a document corpus, embed a whole catalog, or run an eval over a huge test set. This is offline LLM inference, and it has the opposite priorities of a chatbot: nobody's waiting on any single result, so throughput and cost matter, not latency.

A batch inference system grinds a huge dataset through models at maximum throughput per dollar. The moves are the inverse of online serving: huge batches to saturate GPUs (latency is free to trade away), a restartable sharded pipeline, and aggressive cost optimization (spot GPUs, batch discounts) — with checkpointing so a job over millions of items survives failures.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end drop checkpointing and reclaim a worker to see what makes a giant job restartable and spot-friendly. Hit Begin.

Step 1 · Not the online API in a loop

Why looping fails

The obvious approach: loop over your dataset and call the online inference API once per item. Why is that a terrible way to process millions of prompts?

Design decision: Loop over millions of items calling the online API one at a time. What's wrong?

The call: It under-utilizes the GPU (tiny batches), hits rate limits, pays latency-optimized prices, and can't restart cleanly mid-job. — The online API is tuned for low latency per request, not throughput, so serial calls leave GPUs mostly idle, run into rate limits, cost far more per token, and a crash partway through a multi-hour job means unclear recovery. Batch inference is a different system optimized for throughput and resilience.

Looping the online API is disastrous at scale: it under-utilizes GPUs (one small request at a time), hits rate limits, pays latency-optimized prices (online serving costs more per token), and has no clean restart if a multi-hour job fails partway. Batch inference is a separate system built for throughput and resilience — the opposite optimization target from online serving.

Throughput ≠ latency: Online serving minimizes time-to-first-token for one user; batch inference maximizes tokens-per-dollar across a whole dataset. Because no one waits on an individual result, you can trade all the per-item latency you want for utilization — a completely different design.

Step 2 · A job, not a request

The batch job model

If it's not per-request, what's the interface? What does a client submit, and what does the system promise?

Model it as a job: the client submits a dataset (millions of inputs) + a prompt template + model choice, and the system returns a job id, processes asynchronously, reports progress, and writes outputs to a results store (each keyed by its input id). There's no synchronous per-item response — you poll or get notified when the job (or a portion) completes. This job abstraction is what lets everything behind it optimize for throughput and survive failures.

Submit → process → collect: A batch job is a long-running, asynchronous unit: dataset in, results out, progress in between. Framing it as a job (not N requests) is what permits sharding, checkpointing, and huge batching — you're optimizing the whole dataset, not each item.

Step 3 · Saturate the GPU

Throughput via big & continuous batching

The core efficiency lever: GPUs are massively parallel and idle most of the time on one request. How do you keep them fully fed when you have millions of prompts and no latency pressure?

Design decision: You have millions of prompts and no latency SLA. How do you maximize GPU throughput?

The call: Process one prompt at a time for accuracy. — One-at-a-time leaves the GPU almost idle — the single biggest waste in inference. Batch inference exists precisely to fill the GPU with parallel work.

Run large batches and continuous (in-flight) batching: feed the GPU as many prompts in parallel as memory allows, and as each sequence finishes, immediately slot in another so the GPU never idles. Because you have no latency SLA, you can push batch sizes far higher than online serving would tolerate — spending latency you don't care about to buy the throughput you do. This is the single biggest lever: a saturated GPU processes many times the tokens-per-dollar of a serial loop.

Continuous batching: Naive batching waits for the whole batch to finish (the slowest sequence stalls the rest). Continuous batching replaces finished sequences on the fly, keeping utilization high despite varying output lengths. Combined with big batches, it's how batch inference extracts maximum throughput per GPU.

Step 4 · Spread the work

A restartable sharded pipeline

Millions of items won't fit through one worker, and a job running for hours will hit failures. How do you parallelize the work and make it survive crashes?

Build a sharded pipeline: a scheduler splits the dataset into shards placed on a durable queue; many workers pull shards, run batched inference, and write outputs to the results store keyed by input id. Because work is sharded through a durable queue and outputs are keyed idempotently, a lost worker's shard is simply retried by another, and the job is restartable — resume the unfinished shards, don't restart the whole thing. Autoscale the worker fleet to the backlog.

Shard + queue + idempotent writes: The classic large-batch-job pattern: partition the dataset, distribute via a durable queue, and write results idempotently by input id. This gives horizontal scale (add workers) and resilience (retry a failed shard) — essential when a job spans millions of items and hours of runtime.

Step 5 · Squeeze each batch

Bucketing & prefix caching

Even with big batches, throughput leaks: prompts vary wildly in length (padding waste), and many prompts share the same long template prefix. How do you extract more?

Two throughput wins. Length bucketing: group prompts of similar length into a batch so you don't pad short sequences up to the longest one (padding is wasted compute) — sort/bucket by token length. Prefix caching: when many prompts share a common prefix (the same instruction/template, few-shot examples, or a shared document), compute that prefix's KV cache once and reuse it across all of them, so you only pay for the differing part. Both exploit structure batch jobs have but online traffic often lacks.

Bucketing + shared prefixes: Batch jobs are uniform in ways online traffic isn't: similar tasks, shared templates. Bucketing removes padding waste; prefix caching removes recomputation of the shared prompt. On a job of millions of similar prompts, these compound into large savings.

Step 6 · Survive hours & spot GPUs

Checkpointing

A ten-million-prompt job runs for hours, and if you run it on cheap spot GPUs (which can be reclaimed anytime), workers will vanish mid-job. How do you not lose (or repay for) work?

Design decision: A long job on interruptible spot GPUs — workers vanish mid-run. How do you not lose or repeat work?

The call: Checkpoint progress (which items are done) and write idempotently, so a reclaimed worker's shard resumes from the last checkpoint. — Record completed items/shards durably and key outputs by input id. When a worker is reclaimed, its shard is reassigned and resumes from the last checkpoint, skipping already-done items — so no work is lost or repeated. This resilience is exactly what makes running on cheap, interruptible spot GPUs safe.

Checkpoint progress: durably record which items/shards are done, and write outputs idempotently by input id so re-processing an item is a harmless no-op. When a worker crashes or a spot instance is reclaimed, its shard is reassigned and resumes from the last checkpoint, skipping completed items — no work lost, none repeated. This turns a giant job into something restartable, which in turn makes it safe to run on cheap, interruptible spot GPUs.

Restartable ⇒ spot-friendly: Checkpointing + idempotent writes = fault tolerance. That fault tolerance is what unlocks the biggest cost lever: spot/preemptible GPUs (often a fraction of on-demand price), which are only usable if your job can survive being interrupted at any moment. Resilience isn't just safety — it's savings.

Step 7 · Grind it cheap

Cost optimization

The whole point of batch is tokens-per-dollar. Beyond batching, what levers cut the cost of a massive inference job?

Stack the cost levers: run on spot/preemptible GPUs (enabled by checkpointing) for a fraction of on-demand price; use provider batch APIs (many offer large discounts for latency-relaxed batch work); right-size the model (a smaller model often suffices for bulk classification/extraction — don't grind ten million prompts through a flagship if a small model passes your eval); schedule for off-peak capacity; and quantize where quality allows. Combined with high GPU utilization and prefix caching, these can cut the bill by an order of magnitude versus a naive online loop.

Every lever trades against latency: Batch's superpower is that you don't need low latency, so you can cash it in everywhere: spot instances, batch-discount APIs, off-peak scheduling, bigger batches, smaller/quantized models. Each would hurt an online service; for a batch job they're free money. Right-sizing the model is often the biggest single win.

Step 8 · The sharp edges

Failures, skew & validation

At the scale of millions, rare problems become certain: transient errors, a few pathological inputs, length skew, and outputs that must be correct in bulk.

Handle failures with per-item retries (with backoff), a dead-letter path for items that keep failing (don't let a handful stall the job), and idempotent writes so retries are safe. Manage skew: a few very long inputs can dominate a batch — bucket by length and cap/split over-long inputs. Validate outputs at scale (schema/format checks, sampling for quality) since you can't eyeball millions — a malformed prompt template can silently ruin the whole run. Track cost and progress per job, and make the pipeline observable so a stuck or degrading job is caught early.

Design for the unhappy path: Transient errors → retry + dead-letter + idempotency. Length skew → bucket + cap. Bulk correctness → schema checks + sampling. At millions of items, "rare" is "constant", so the pipeline must self-heal around bad inputs and validate output quality automatically — a broken template shouldn't cost you a million wasted prompts unnoticed.

You did it

You just designed a batch inference system.

• B — a
• L — o
• M — o
• S — a
• S — h
• S — q
• C — h