The whole design, in writing
Learn AI system design by building an offline batch LLM inference system that processes millions of prompts step by step. An interactive guide covering why looping the online API fails, optimizing throughput over latency, a restartable sharded pipeline, batching techniques (continuous batching, length bucketing, prefix caching), scaling on spot GPUs with checkpointing, cost optimization, and the unhappy paths (failures at scale, idempotency, skew).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is batch inference?
Sometimes you don't need an answer now — you need millions of answers, cheaply, by tomorrow: classify every support ticket, summarize a document corpus, embed a whole catalog, or run an eval over a huge test set. This is offline LLM inference, and it has the opposite priorities of a chatbot: nobody's waiting on any single result, so throughput and cost matter, not latency.
A batch inference system grinds a huge dataset through models at maximum throughput per dollar. The moves are the inverse of online serving: huge batches to saturate GPUs (latency is free to trade away), a restartable sharded pipeline, and aggressive cost optimization (spot GPUs, batch discounts) — with checkpointing so a job over millions of items survives failures.
What the new pieces do
- Clientclient
- Submits a big offline job — classify/summarize/embed millions of documents, or run an eval — where throughput matters, not per-item latency.
Step 1 · Not the online API in a loop
Why looping fails
The obvious approach: loop over your dataset and call the online inference API once per item. Why is that a terrible way to process millions of prompts?
Loop over millions of items calling the online API one at a time. What's wrong?
Simple but disastrous at scale: it under-utilizes GPUs (tiny per-request batches), hits rate limits, pays online (latency-optimized) prices, and offers no restartability for a job that runs for hours over millions of items.
Answers are fine; efficiency is the problem. One-at-a-time calls waste GPU throughput, get rate-limited, cost more, and can't cleanly resume after a failure mid-job.
The online API is tuned for low latency per request, not throughput, so serial calls leave GPUs mostly idle, run into rate limits, cost far more per token, and a crash partway through a multi-hour job means unclear recovery. Batch inference is a different system optimized for throughput and resilience.
Looping the online API is disastrous at scale: it under-utilizes GPUs (one small request at a time), hits rate limits, pays latency-optimized prices (online serving costs more per token), and has no clean restart if a multi-hour job fails partway. Batch inference is a separate system built for throughput and resilience — the opposite optimization target from online serving.
Why this piece earns its place
The repair everyone reaches for first is to fan the loop out — a thread pool, many calls in flight — and it is worth being able to say why that still does not get you there. Concurrency at the client is not batching at the server. Each of those calls arrives on its own and is scheduled on its own; the provider may coalesce them for its own efficiency, but it does that inside a latency target it owes other people, so the one number that decides the economics here — how many prompts sit in a single forward pass — is set by somebody optimizing for the opposite thing. You have bought parallelism at the wrong layer. The loop also has nowhere to keep its progress. That progress is a variable inside a running process, so there is nothing for a second attempt to read, and we will make it restartable later turns out to be a rewrite rather than a patch. There is a real line where the loop is still the right call, though. If starting over is annoying rather than expensive, a script with a resume file beats a job system you now have to operate. The threshold is not item count — it is whether the run outlives the terminal it was started from.
Step 2 · A job, not a request
The batch job model
If it's not per-request, what's the interface? What does a client submit, and what does the system promise?
Model it as a job: the client submits a dataset (millions of inputs) + a prompt template + model choice, and the system returns a job id, processes asynchronously, reports progress, and writes outputs to a results store (each keyed by its input id). There's no synchronous per-item response — you poll or get notified when the job (or a portion) completes. This job abstraction is what lets everything behind it optimize for throughput and survive failures.
Why this piece earns its place
Look at what the client can actually hand over. Millions of inputs do not travel in a request body, so the dataset arrives as a reference — a path, a table name, a prefix in object storage — and the job then reads that reference for hours. It is mutable, and nothing in the diagram says otherwise. Rows appended mid-run are processed or skipped depending on where a worker had got to; rows edited in place get whichever version the worker happened to read, and a retried shard can produce the other one; a file replaced at hour three fails shards that were fine at hour one. So the submit path has to resolve and freeze the dataset — an immutable snapshot, or at minimum a version id and an item count the scheduler shards against — because everything downstream assumes the work is a fixed set. Two smaller obligations come with the job model. Cancel is a real terminal state, not a UI affordance: somebody will submit the wrong dataset, and a job whose only ending is finishing will keep buying GPU hours to write output nobody will read, so cancel has to reach the workers. And the job id is the only usable cost-attribution key once several teams share one fleet — what did that experiment cost is unanswerable per instance and answerable per job.
- jobdataset + template
- asyncprogress + job id
- resultskeyed by input id
What the new pieces do
- Batch APIgateway
- Accepts a dataset + prompt template, kicks off a job, and reports progress — no synchronous per-item response.
Step 3 · Saturate the GPU
Throughput via big & continuous batching
The core efficiency lever: GPUs are massively parallel and idle most of the time on one request. How do you keep them fully fed when you have millions of prompts and no latency pressure?
You have millions of prompts and no latency SLA. How do you maximize GPU throughput?
With no per-item latency requirement, you feed the GPU as many prompts in parallel as memory allows (large batches), and use continuous/in-flight batching so finished sequences are immediately replaced by new ones — keeping the GPU near 100% utilization. You're spending latency you don't care about to buy throughput you do.
One-at-a-time leaves the GPU almost idle — the single biggest waste in inference. Batch inference exists precisely to fill the GPU with parallel work.
Hardware choice aside, the throughput win comes from batching work onto each GPU. Without large/continuous batching, even more GPUs stay underutilized.
Run large batches and continuous (in-flight) batching: feed the GPU as many prompts in parallel as memory allows, and as each sequence finishes, immediately slot in another so the GPU never idles. Because you have no latency SLA, you can push batch sizes far higher than online serving would tolerate — spending latency you don't care about to buy the throughput you do. This is the single biggest lever: a saturated GPU processes many times the tokens-per-dollar of a serial loop.
Why this piece earns its place
Two things here you only learn by running it. First, the batch size is a memory budget, not a number. What fills the card is the KV cache, and that grows with sequence length times concurrency, so the largest safe batch depends on what happens to be in it. Cap it by tokens in flight rather than by request count, because setting it too high does not make the worker slow — it gets the process killed partway through a shard, and with continuous batching that kill takes every sequence currently in flight with it, not one. Second, the near-100% utilization this step is chasing is a floor check, not a tuning signal. It tells you the worker is batching at all, which a serial loop never manages, and then it stops moving as a number long before the tuning is done. What the driver reports is that a kernel was resident, not that the hardware did useful work, so a worker can sit pinned there while it shuffles memory or computes padding — which is how a throughput regression survives a week of green dashboards. Once you are tuning rather than checking, the number that answers the question is output tokens per GPU-second against a fixed sample.
- big batchesfill GPU memory
- continuousrefill finished slots
- ~100%GPU utilization
Step 4 · Spread the work
A restartable sharded pipeline
Millions of items won't fit through one worker, and a job running for hours will hit failures. How do you parallelize the work and make it survive crashes?
Build a sharded pipeline: a scheduler splits the dataset into shards placed on a durable queue; many workers pull shards, run batched inference, and write outputs to the results store keyed by input id. Because work is sharded through a durable queue and outputs are keyed idempotently, a lost worker's shard is simply retried by another, and the job is restartable — resume the unfinished shards, don't restart the whole thing. Autoscale the worker fleet to the backlog.
Why this piece earns its place
A worker here is not a task, and that carries further than it looks. Loading the model is the expensive part of starting one — the weights have to be pulled and made resident in GPU memory before a single prompt is processed, and on anything worth batching that dominates startup — so a worker is a long-lived process that takes shard after shard, never a process spawned per shard. That bends the scaling policy away from anything a web tier does. Scale-up is measured in minutes rather than the seconds an HTTP autoscaler assumes, so you scale on the trend of the backlog instead of its depth at this instant, and you keep a worker warm between shards rather than handing it back, because a fleet that scales down eagerly pays that load time again on the way back up. It also changes what a failed shard means. Because one worker takes many shards in a row, a host with a sick GPU, a half-written model directory or a full disk fails everything it touches, and a shard returned to the queue in front of a fast-draining fleet often lands straight back on the machine that just failed it. Count failures per worker as well as per shard, and take a host out of the fleet when it trips — otherwise one bad machine quietly eats the run.
- shardpartition the dataset
- workerspull + process in parallel
- idempotentwrite by input id
What the new pieces do
- Work Queuebus
- Durably holds the job's work items in shards, so many workers pull in parallel and a lost worker's shard is retried.
- Schedulerservice
- Splits the dataset into shards, tracks progress, and scales the worker fleet (often on spot GPUs) to the backlog.
- Inference Workerservice
- Pulls a shard and runs the model over it in LARGE batches to saturate the GPU — throughput over latency.
- Results Storedata
- Where outputs land, keyed by input id so results are idempotent and the job is resumable.
Step 5 · Squeeze each batch
Bucketing & prefix caching
Even with big batches, throughput leaks: prompts vary wildly in length (padding waste), and many prompts share the same long template prefix. How do you extract more?
Two throughput wins. Length bucketing: group prompts of similar length into a batch so you don't pad short sequences up to the longest one (padding is wasted compute) — sort/bucket by token length. Prefix caching: when many prompts share a common prefix (the same instruction/template, few-shot examples, or a shared document), compute that prefix's KV cache once and reuse it across all of them, so you only pay for the differing part. Both exploit structure batch jobs have but online traffic often lacks.
Why this piece earns its place
Both of these have a consequence that lands somewhere else in the design. Bucketing reorders the dataset: results stop arriving in input order, permanently, and anything downstream that assumed a stable order is now wrong in a way no test on a small file will catch. That is what makes keying every write by input id non-negotiable rather than merely tidy. Prefix caching has a sharper edge. The cache is keyed on the exact token prefix, so one character of drift breaks it — a trailing space, a date rendered into the system prompt, or a per-item field someone placed above the shared instructions instead of below them. Nothing errors and nothing logs; you quietly pay the full prompt for every item in the run. So prompt layout is a performance decision: everything shared goes first, everything per-item goes last, and the template is a file you diff rather than a string you edit in place. Skip bucketing entirely if the prompts are already uniform — a fixed-form classification job gains almost nothing from it, and you have added a sort pass over a dataset that does not fit in memory.
- bucket by lengthcut padding waste
- prefix cachecompute shared prompt once
- reuseacross the batch
What the new pieces do
- Modeldata
- The loaded model, fed as many prompts in parallel as fit — continuous batching keeps it fully utilized.
Step 6 · Survive hours & spot GPUs
Checkpointing
A ten-million-prompt job runs for hours, and if you run it on cheap spot GPUs (which can be reclaimed anytime), workers will vanish mid-job. How do you not lose (or repay for) work?
A long job on interruptible spot GPUs — workers vanish mid-run. How do you not lose or repeat work?
That works but throws away the biggest cost saving (spot GPUs are far cheaper). With checkpointing, batch jobs tolerate interruptions, so you can use cheap spot capacity.
Restarting from zero after processing millions of items wastes enormous compute and money. You checkpoint progress so you resume from where you were, not the beginning.
Record completed items/shards durably and key outputs by input id. When a worker is reclaimed, its shard is reassigned and resumes from the last checkpoint, skipping already-done items — so no work is lost or repeated. This resilience is exactly what makes running on cheap, interruptible spot GPUs safe.
Checkpoint progress: durably record which items/shards are done, and write outputs idempotently by input id so re-processing an item is a harmless no-op. When a worker crashes or a spot instance is reclaimed, its shard is reassigned and resumes from the last checkpoint, skipping completed items — no work lost, none repeated. This turns a giant job into something restartable, which in turn makes it safe to run on cheap, interruptible spot GPUs.
Why this piece earns its place
The knob here is granularity, and it is a pricing decision rather than a correctness one. Checkpointing every item is exact and turns one write into two, putting a second round trip on the hot path of every single prompt. Checkpointing every N items means a resume may redo up to N, which is harmless because the writes are keyed by input id — so the only thing you lose is money. Size N by what you are willing to pay for twice, not by what feels safe. Two things follow. The checkpoint store ends up the highest-write-rate component in the whole system, taking small updates continuously from every worker, and if it shares a database with the results it competes with the thing you actually care about. And “re-running an item is a no-op” holds only while generation is deterministic: with any sampling temperature the redo produces a different valid answer, so a resumed job’s outputs are not one consistent sample of the model. Fine for bulk extraction, not fine for an eval, where you pin the seed or record which rows were written on which pass.
- checkpointdone items/shards
- idempotentre-run = no-op
- spot GPUscheap, interruptible
What the new pieces do
- Checkpointdata
- Records which items are done so a crashed/reclaimed worker resumes instead of reprocessing millions of prompts.
Step 7 · Grind it cheap
Cost optimization
The whole point of batch is tokens-per-dollar. Beyond batching, what levers cut the cost of a massive inference job?
Stack the cost levers: run on spot/preemptible GPUs (enabled by checkpointing) for a fraction of on-demand price; use provider batch APIs (many offer large discounts for latency-relaxed batch work); right-size the model (a smaller model often suffices for bulk classification/extraction — don't grind ten million prompts through a flagship if a small model passes your eval); schedule for off-peak capacity; and quantize where quality allows. Combined with high GPU utilization and prefix caching, these can cut the bill by an order of magnitude versus a naive online loop.
Why this piece earns its place
The largest cost line on a job like this is often not infrastructure at all — it is the output tokens you asked for. Generation is the expensive direction, and a template that tells the model to explain its reasoning before giving a label multiplies output length across every item in the dataset. Capping the response and asking for a bare label beats every lever above it, and it is a prompt change rather than a platform change, which means it is available on the first day. The second thing to say out loud is that these levers do not stack the way a list implies. A provider batch endpoint and your own spot fleet are alternatives, not additions: you cannot claim their discount on hardware you rent yourself, and inside their endpoint the quantization, the GPU generation and the batch size are theirs to choose. The decision underneath is buy versus run, and it turns on whether the scarce resource is GPU spend or the engineering time to operate a fleet. And when you test the smaller model, test it against labelled data, not against the flagship output. Scoring one model on its agreement with another quietly promotes the flagship to ground truth and hides every case where the two are wrong together.
- spot GPUsfraction of on-demand
- batch APIdiscounted
- right-sizesmaller model if it passes
Step 8 · The sharp edges
Failures, skew & validation
At the scale of millions, rare problems become certain: transient errors, a few pathological inputs, length skew, and outputs that must be correct in bulk.
Handle failures with per-item retries (with backoff), a dead-letter path for items that keep failing (don't let a handful stall the job), and idempotent writes so retries are safe. Manage skew: a few very long inputs can dominate a batch — bucket by length and cap/split over-long inputs. Validate outputs at scale (schema/format checks, sampling for quality) since you can't eyeball millions — a malformed prompt template can silently ruin the whole run. Track cost and progress per job, and make the pipeline observable so a stuck or degrading job is caught early.
Why this piece earns its place
The retry budget has to be global, not per item. Per-item backoff absorbs a flaky minute, but a systematic failure — an expired credential, an endpoint rejecting one field of the template — fails every item identically, and a per-item policy will cheerfully re-pay for millions of rejections at full fleet throughput before anyone looks. Cap the total and halt the job when it trips. The dead-letter path needs the same treatment, because it is a queue nobody opens unless something alarms on its rate: a steady trickle is a slice of the dataset silently missing from a job that reports itself finished, and downstream nothing distinguishes a row that failed from a row never asked for. The subtler one is that schema checks pass on confidently useless output. Valid JSON carrying the label unknown on a large share of rows is green on every automated check you wrote. So alarm on the distribution of outputs against a pilot you ran before the full job — and run that pilot. A small sample through the exact template and model costs nothing next to the run it protects.
You did it
You just designed a batch inference system.
Everything you assembled, in order
- Batch inference is offline, throughput-first — the opposite priorities of latency-sensitive online serving.
- Looping the online API fails: idle GPUs, rate limits, high per-token cost, no clean restart.
- Model it as an async job (dataset + template in, results by input id out), not per-request calls.
- Saturate GPUs with huge + continuous batching — trade the latency you don't need for throughput.
- Shard the dataset through a durable queue to many workers with idempotent, resumable writes.
- Squeeze more with length bucketing (cut padding) and prefix caching (compute shared prompts once).
- Checkpoint progress so jobs are restartable — which unlocks cheap spot GPUs, batch-discount APIs and right-sizing.
