KV Cache Calculator
Pick a model architecture (GQA-aware presets from published configs), a cache dtype (FP16/FP8/INT4) and a context length, and see the KV cache laid bare: bytes per token, gigabytes per sequence, the whole batch — and how many full-context users actually fit beside the weights on a 24/48/80/141 GB card.
The per-user memory tax
Weights are a one-time cost — load the model once, every user shares it. The KV cache is per user: every in-flight request stores the attention keys and values for every token of its context, for its entire lifetime. That is the tax this calculator measures: 2 × layers × kv-dim × dtype bytes per token, where kv-dim is the hidden size scaled down by the GQA ratio.
GQA: the reason long context is servable
Flip between the Llama presets and the legacy no-GQA model. Grouped-query attention shares each K/V head across many query heads — 8 KV heads under 64 query heads is an 8× cache reduction with negligible quality cost. Before GQA, a 13B model at long context could out-eat its own weights in cache; after it, 70B models serve 128K contexts on single nodes. One architecture choice, an order of magnitude of serving economics.
Why this number caps concurrency
Decode is memory-bound: per request, the GPU's compute units are mostly idle. So what stops you adding another user is never FLOPs — it is whether another full-context cache fits in the VRAM left beside the weights. The fits-table makes it concrete: at 128K context, a 24 GB card that happily holds an 8B model may fit only a couple of concurrent long-context users. That cliff is why providers price long context so aggressively.
Squeezing the cache
Three production levers, in order of adoption: FP8 cache (halves it, near-free quality-wise, now standard), paged attention (doesn't shrink it, but stops fragmentation from wasting what you have), and INT4 cache (quarters it — powerful for extreme contexts, but quantization error compounds over long generations, so eval before trusting). Prefix sharing goes further still: identical system prompts across users can share one cached prefix.
How it works
- Per token = 2 × layers × (hidden × kv-heads ÷ heads) × dtype bytes.
- GQA presets use published configs; the legacy preset shows no-GQA pain.
- Fits-table: 90% usable VRAM, weights held FP16.
- FP8 cache ≈ 2× concurrency; INT4 ≈ 4× — validate quality per task.
Frequently asked questions
How is KV cache size computed?
Per token: 2 (keys + values) × layers × kv-dimension × dtype bytes, where kv-dimension = hidden size × kv-heads ÷ attention-heads. Multiply by context length for a sequence, and by concurrent sequences for the batch. The presets use each model’s published architecture numbers.
Why does grouped-query attention (GQA) shrink the cache so much?
GQA shares each key/value head across several query heads — Llama-70B runs 8 KV heads under 64 query heads, an 8× cache reduction. It is the single biggest reason modern models are servable at long context; the legacy no-GQA preset shows what life used to cost.
Why does the KV cache cap concurrency instead of compute?
Decode barely uses the GPU’s compute per request, but every in-flight request holds its full cache in VRAM for its whole lifetime. The weights are paid once; the cache is paid per user. When the free VRAM beside the weights is spent, admission stops — regardless of idle FLOPs.
Is quantizing the KV cache safe?
FP8 cache is widely used in production with negligible quality loss, roughly doubling concurrency. INT4 cache is more aggressive — helpful for very long contexts, but validate on your task: cache quantization errors compound over long generations in a way weight quantization does not.