Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
KV Cache Calculator
Pick a model architecture (GQA-aware presets from published configs), a cache dtype (FP16/FP8/INT4) and a context length, and see the KV cache laid bare: bytes per token, gigabytes per sequence, the whole batch — and how many full-context users actually fit beside the weights on a 24/48/80/141 GB card.
Weights are a one-time cost: load the model once and every user shares it. The KV cache is paid per request. Each in-flight sequence stores the attention keys and values for every token of its context for as long as it runs. This calculator measures that per-request memory and shows how many full-context requests fit beside the weights on common GPUs.
What it computes
kv_dim = hidden × kv_heads / heads
per token = 2 × layers × kv_dim × bytes 2 = one key + one value
per seq = per token × context / 1,024³ GB
batch = per seq × concurrent sequences
fits(card) = ⌊ (card × 0.9 − FP16 weights) / per seq ⌋- Layers, hidden, heads, kv_heads come from each preset’s published configuration. The ratio
heads / kv_headsis the grouped-query attention saving shown under the model picker. - Bytes is the cache data type: 2 for FP16, 1 for FP8 and 0.5 for INT4.
- Cache vs weights divides the batch total by the model’s approximate FP16 weight size.
- The fits table assumes 90% of each card is usable and the weights stay in FP16; if the weights alone do not fit, it says so.
A worked example
The defaults are Llama 3.1 8B (32 layers, hidden size 4,096, 32 heads, 8 KV heads), an FP16 cache, a 32,768-token context and 8 concurrent sequences.
- KV dimension:
4,096 × 8 / 32 = 1,024; GQA saves32 / 8 = 4×. - Per token:
2 × 32 × 1,024 × 2 = 131,072 bytes, shown as 128 KB. - Per sequence:
131,072 × 32,768 = 4 GBexactly. - Whole batch:
4 × 8 = 32 GB, which is32 / 15 = 213%of the weights.
| Card | Usable | Free beside 15 GB weights | 32K sequences that fit |
|---|---|---|---|
| 24 GB | 21.6 GB | 6.6 GB | 1 |
| 48 GB | 43.2 GB | 28.2 GB | 7 |
| 80 GB | 72 GB | 57 GB | 14 |
| 141 GB | 126.9 GB | 111.9 GB | 27 |
Switch the cache to FP8 and every row roughly doubles, to 3, 14, 28 and 55. Switch to the legacy 13B model without GQA and the cache grows to 800 KB per token and 25 GB per sequence, so even an 80 GB card holds just one.
Assumptions and limits
- Every sequence at full context. Real requests vary in length, and paged attention allocates cache in small blocks as tokens arrive, so average use is lower than this worst case.
- Weights fixed at FP16. The fits table ignores weight quantization; an INT4 model leaves far more room for cache.
- No other memory. Activations, CUDA workspaces and fragmentation are covered only by the 10% held back.
- Standard attention layout. Models with sliding-window attention, compressed latent attention or cross-layer sharing need less than this formula gives.
How to use the result
- Turn it into an admission limit. The fits number is roughly the maximum concurrent full-length requests before the server must queue.
- Pull the cheapest lever first. FP8 cache halves memory with little quality cost in most engines; then cap context length per tier of service.
- Share prefixes. Requests with the same system prompt can share its cached tokens, which matters most when prompts are long.
- Price long context deliberately. At 128K, one 8B sequence costs 16 GB of cache, so a long-context user can displace several short ones.
How it works
- Per token = 2 × layers × (hidden × kv-heads ÷ heads) × dtype bytes.
- GQA presets use published configs; the legacy preset shows no-GQA pain.
- Fits table: 90% usable VRAM, weights held FP16.
- FP8 cache ≈ 2× concurrency; INT4 ≈ 4×; validate quality per task.
Frequently asked questions
How is KV cache size computed?
Per token it is 2, for keys and values, times the number of layers, times the KV dimension, times the bytes per value. The KV dimension is the hidden size times the number of KV heads divided by the number of attention heads. Multiply by the context length for one sequence and by the number of concurrent sequences for the batch. For Llama 3.1 8B in FP16 that is 128 KB per token, 4 GB for a 32K-token sequence and 32 GB for eight of them.
Why does grouped-query attention (GQA) shrink the cache so much?
GQA lets several query heads share one key-value head, so the cache stores only the KV heads. Llama 3.1 70B runs 8 KV heads under 64 query heads, an 8 times reduction, and Llama 3.1 8B runs 8 under 32, a 4 times reduction. Models without GQA store a key and value for every head: the legacy 13B preset needs 800 KB per token, more than six times the 8B model’s 128 KB, despite having less than twice the parameters.
Why does the KV cache cap concurrency instead of compute?
During generation each request does little arithmetic per step but keeps its whole cache in GPU memory for its entire lifetime. The weights are paid once and shared, while the cache is paid per request. When the memory left beside the weights is used up, the server must queue new requests no matter how idle the compute units are. In the default example a 24 GB card holds the 8B model’s weights but only one 32K-token sequence beside them.
How much KV cache does a 128K context need?
For Llama 3.1 8B in FP16 the cache is 128 KB per token, so a 131,072-token sequence needs 16 GB, slightly more than the model’s roughly 15 GB of FP16 weights. Llama 3.1 70B needs 320 KB per token, or 40 GB per 128K sequence. That is why long-context serving caps the number of simultaneous users so hard, and why FP8 cache, paged allocation and prefix sharing matter most at long context.
Is quantizing the KV cache safe?
FP8 cache is widely used in production serving engines and usually costs little quality, while halving the cache and roughly doubling how many sequences fit. INT4 cache quarters it and helps at very long contexts, but quantization errors in the cache can accumulate over long generations in a way weight quantization does not, so validate it on your own task and context lengths before relying on it. Always compare outputs at your real prompt lengths, not just short tests.