A model generating text pays a hidden rent: for every token, in every layer, it stores a key and a value so it never recomputes the past. Watch that cache stack up, collapse it with grouped-query attention, slam into the VRAM wall that caps how many users a GPU serves — then quantize it back.
Act 1 · Prefill & decode
Watch the cache grow
Feed a prompt (prefill), then generate tokens one at a time (decode). Each token appends a key and value to the cache — for every layer. Nothing is recomputed; it just accumulates.
Sequence (KV cached per token)
0
Tokens cached
0
K/V vectors stored
across 32 layers
0×
Recompute avoided
vs no-cache
Prefill processes the whole prompt in parallel; decode appends one token per step. Watch the "recompute avoided" climb — without the cache, step N would re-run attention over all N tokens.
Act 2 · Grouped-query attention
Collapse the cache with GQA
Standard attention (MHA) caches a separate key/value for every query head. GQA lets several query heads share one K/V head. Fewer K/V heads = smaller cache — the trick that made long context servable.
Query heads32
KV heads (GQA group)8
Query heads → shared KV heads
4×
Cache reduction
vs full multi-head
4:1
Query : KV heads
— KB
Per token / seq
llama-70B geometry
GQA with 8 KV heads under 64 query heads is an 8× cache cut — for negligible quality loss. Drag KV heads to 1 for multi-query attention (maximum sharing), or up to match query heads for full multi-head.
Act 3 · The concurrency wall
Where the cache caps your users
Weights load once and everyone shares them. But every concurrent user holds their ownKV cache for their whole context. Push context and batch up and watch the cache eat the free VRAM — then overflow it.
Context length / user8K
Concurrent users8
GPU:
weights 132 GB (shared)cache 0 GB
0
Max users that fit
at this context
0 GB
Cache demanded
0 GB
Free beside weights
Note what caps you: not compute, but whether another cache fits. This is why providers meter long context so aggressively — a 128K request can cost as much cache as dozens of short ones.
Act 4 · Quantize the cache
Claw the memory back
The cache doesn't have to be FP16. Store keys and values in fewer bits and it shrinks proportionally — buying more concurrent users from the same card, at some risk to quality on long generations.
Cache dtype:
1×
Cache shrink
vs FP16
—
Users on 80 GB
70B, 32K context
safe
Quality risk
FP8 is the production sweet spot — roughly 2× the users, negligible quality loss. INT4 quarters the cache but its error compounds over long outputs, so validate it on your task before shipping.
Why the cache exists
A transformer generates one token at a time, and each new token must attend to every previous token. Done naively, generating token N re-runs attention over all N tokens — quadratic re-work that gets slower every step. The fix is the KV cache: after processing a token, keep its attention keys and values around. The next step only computes the new token's query and reuses every cached K and V. Generation becomes a linear append.
The trade
You trade compute for memory. The math per token drops enormously; in exchange you store, for every token × every layer × every sequence in flight, a key and a value. That storage is the KV cache — and it grows without mercy.
The size formula
Per token, the cache is 2 × layers × (hidden × kv_heads ÷ heads) × dtype_bytes — the 2 is keys plus values, and the kv_heads ÷ heads ratio is exactly what GQA shrinks. Multiply by context length for one sequence, and by concurrent sequences for the batch. For a 70B model at long context, this reaches tens of gigabytes — rivaling the weights.
Why it rules serving
Decode is memory-bound: the GPU's compute sits mostly idle while it streams weights and cache. So what stops you adding another user is never spare FLOPs — it's whether another full cache fits in the VRAM left beside the weights. The KV cache, not compute, is the concurrency ceiling, which is why every serving optimization (GQA, paged attention, quantized caches, prefix sharing) is really a cache-shrinking trick.
The KV cache, step by step
What happens to the cache during one request, the byte count for real model shapes, and how many users those bytes allow on one GPU.
The mechanism
Prefill. Run the whole prompt through the model in parallel and store every token's key and value in every layer.
Decode one token. Compute the query, key and value for the new token only, and append its key and value to each layer's cache.
Attend. The new query scores against all t cached keys and mixes their values: work proportional to t per layer, instead of recomputing keys and values for all t tokens again.
Release. When the request ends, its cache is freed. Paged attention keeps the cache in fixed-size blocks allocated as the sequence grows, so nothing is reserved for a maximum length up front.
Per token the cache holds 2 × layers × kv_heads × head_dim × bytes_per_value bytes: the formula above with head_dim = hidden ÷ heads. Multiply by tokens for a sequence and by sequences for the batch.
Worked example: real shapes
Llama 3 8B: 32 layers, 8 KV heads, 128 dimensions, 2 bytes in bf16. 2 × 32 × 8 × 128 × 2 = 131,072 bytes, 128 KiB per token, so an 8,192-token conversation holds exactly 1 GiB.
Without GQA, the same shape with 32 KV heads (Llama 2 7B's layout) needs 512 KiB per token and 4 GiB per 8K sequence.
Llama 3 70B: 80 layers, 8 KV heads, 128 dimensions gives 320 KiB per token, the number Act 2 shows: 2.5 GiB at 8K and 40 GiB at 128K. With all 64 heads cached it would be 2.5 MiB per token.
Concurrency. On an 80 GB card with 90% usable (72 GiB), the 8B model's 15 GiB of weights leave 57 GiB of cache: 57 users at 8K context, or 3 at 128K. An FP8 cache doubles that to 114 and 7.
The 70B model. Its 132 GiB of 16-bit weights exceed 90% of even a 141 GB card (126.9 GiB) before a single token is cached. Serving it on one card takes quantized weights (about 35 GB at 4 bits); otherwise the weights are split across several GPUs with tensor parallelism.
Knobs and pitfalls
Size memory for peak concurrency times peak context, not the average, and leave room for activations; serving engines preallocate the cache for this reason (vLLM claims 90% of GPU memory by default). Prefix caching stores a shared system prompt once for every request that uses it. Sliding-window attention caps each sequence's cache at the window length, and DeepSeek's multi-head latent attention caches a compressed latent instead of full keys and values. Quantizing the cache to FP8 or lower is the cheapest win, but check accuracy on long-context tasks, where small errors in old keys are read on every later step.
Check yourself
Four questions. The cache remembers if you don't.
See every answer, with the reasoning
Why does a transformer cache keys and values during generation?
Answer: So each new token doesn’t recompute attention over the whole past — linear append instead of quadratic re-work — The cache trades memory for compute: store past K/V once, reuse them every step.
What does grouped-query attention shrink?
Answer: The KV cache — by sharing K/V heads across query heads — 8 KV heads under 64 query heads is an 8× cache cut, for negligible quality loss.
On an inference GPU, what usually caps how many users you can serve?
Answer: The KV cache filling the VRAM left beside the weights — Decode is memory-bound; the ceiling is whether another cache fits, not spare compute.
What does moving the KV cache from FP16 to FP8 buy?
Answer: Roughly 2× the concurrent users, negligible quality loss — Half the bytes per K/V = half the cache = about double the users. FP8 is the production default.