Why is serving an LLM hard?

Treat the GPU fleet as a precious, fixed resource and design everything around it: a queue to absorb spikes, continuous batching to keep GPUs full, a KV cache to make each token cheap, sharding to fit big models, and autoscaling to ride demand.

A client, a router, a model

A Router fronts the fleet: it admits requests, applies rate limits, and routes each to a GPU Worker with capacity, then streams the generated tokens back to the Client. The classic client → router → workers spine, tuned for streaming.

Put a queue in front of the GPUs

A Request Queue sits between the router and the GPUs. Bursts fill the queue instead of crushing the workers; in quiet moments the queue drains and GPUs stay busy. Admission control can shed load ("busy, retry") when the queue grows too deep — protecting latency for everyone already in flight.

The GPU Workers do two phases. Prefill processes the entire prompt in one parallel pass (compute-bound). Decode then generates output one token per step, each step depending on the last (memory-bandwidth-bound). The asymmetry — fast parallel prefill, slow sequential decode — shapes every serving decision.

A Batch Scheduler does continuous (in-flight) batching: at each decode step it runs all active sequences together, retires the ones that just finished, and slots waiting requests in immediately — no waiting for a batch to drain. The GPU stays full no matter how lengths and arrivals mix. Serving metrics (TTFT, tokens/sec, utilization) drive its decisions.

The KV cache and paged attention

The KV Cache stores each token’s attention key/value state so every new token reuses prior work instead of recomputing — turning decode from quadratic to linear. Paged attention stores that cache in fixed-size blocks (like OS virtual memory), so many sequences share GPU memory without waste — which is exactly what lets continuous batching pack so many requests in.

Shard the weights across GPUs

The Model Weights are tensor-sharded across several GPUs: each layer’s matrices are split so the GPUs compute a forward pass together over fast interconnect, acting as one logical worker. Even larger models add pipeline parallelism (split by layer). The fleet is then many such multi-GPU workers behind the scheduler.

Autoscale on GPU pressure

An Autoscaler watches the true pressure signals — queue depth and GPU utilization — and adds or removes replicas to hold latency targets. Because GPUs are slow to start, it keeps a warm pool and scales ahead of the curve. Under extreme load it sheds or routes overflow to a smaller, faster model.

Design an LLM Inference Server — A Guided System Design

System Design · step by stepDesign an LLM Inference Server

Step 1 / 9

Finished this one? 0 / 5 AI System Designs done

More AI System Designs

Design a Conversational AIBuild a production conversational AI system (think ChatGPT) step by step. See how the request path splits an inference gateway from the model servers, how the context window is assembled and token-budgeted, how conversation memory is stored and recalled, how tokens stream back over a persistent connection, and how guardrails gate every prompt and response — through an interactive diagram that grows with each concept.Read →
Design a RAG PipelineBuild a retrieval-augmented generation pipeline step by step. See how documents are chunked and embedded, how a vector store answers semantic search, how two-stage retrieval with reranking finds the best passages, how the prompt is grounded to stop hallucination, and how evals keep a quietly-drifting index honest — through an interactive diagram that grows with each concept.Read →
Design an AI Agent SystemBuild an autonomous AI agent step by step. See how the plan-act-observe loop turns a goal into action, how the model emits typed tool calls, how a sandboxed executor runs them safely, how working and long-term memory fit together, and how budgets and approval gates keep a multi-step agent from running away — through an interactive diagram that grows with each concept.Read →
Design a Recommendation SystemBuild a large-scale recommendation system step by step. See how a two-stage retrieve-and-rank funnel picks the best few from millions, how two-tower embeddings and ANN generate candidates fast, how a heavy ranking model scores engagement, how a feature store stays consistent between training and serving, and how the feedback loop keeps recommendations fresh — through an interactive diagram that grows with each concept.Read →

Explore more from Vibe Engines

Handbooks →

Roadmaps →

Prompt Engineering Roadmap

System Designs →