Inference & Serving
All 12 cards in this deck
KV cache, batching, quantization, TTFT — how tokens actually get made, fast and cheap. Answer each question before you open it — then run the review above, which schedules every card with spaced repetition so it comes back right before you would forget it.
What are the two phases of LLM inference?
Prefill (process the prompt, build KV state — compute-bound) and decode (emit tokens one by one — memory-bandwidth-bound).
What does the KV cache store, and why?
Every previous token’s attention keys/values, so each new token doesn’t re-process the whole prompt — linear generation instead of quadratic re-work.
What usually caps concurrency on an inference GPU?
KV-cache memory, not compute — every in-flight request holds KV state proportional to its context length.
What is continuous batching?
Every forward pass advances all in-flight requests one token, and new arrivals join freed slots mid-flight — no waiting for the batch to drain.
What is TTFT and why does it matter most in chat?
Time to first token (queueing + prefill). Once tokens stream users read along — the blank-box wait before the first token is perceived latency.
What does prefix/prompt caching attack — prefill or decode?
Prefill: cached KV state of unchanged prefixes means the GPU only processes genuinely new tokens. Batching attacks decode.
INT8 and INT4 quantization cut weight memory by…?
2× and 4× vs FP16 (2 bytes/param). Note: the KV cache typically stays FP16, so quantization shrinks weights, not context memory.
How does speculative decoding speed things up without changing outputs?
A small draft model proposes several tokens; the big model verifies them in one pass, keeping the longest correct prefix — provably identical distribution.
Quantization vs distillation?
Quantization stores the same weights in fewer bits; distillation trains a new smaller model to imitate the big one.
What is grouped-query attention (GQA) for?
Sharing KV heads across query heads cuts the KV cache 4–8× — the main reason modern models are servable at long context.
Rough full fine-tune memory per parameter with AdamW?
~16 bytes/param (weights + gradients + two optimizer moments) before activations — why LoRA/QLoRA exist.
What is a model router and why is it the #1 cost lever?
A classifier sending easy prompts to small models, hard ones to frontier models — routing half the traffic to a 10×-cheaper model roughly halves the GPU bill.