Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
GPU VRAM Calculator
Pick a model (Llama, Mistral, Qwen, Phi, Mixtral or custom architecture), a precision (FP16 / INT8 / INT4), a context length and batch size — and get the real VRAM requirement: weights, KV cache (GQA-aware), runtime overhead, and a fits-or-not verdict across common GPUs from a T4 to an H200.
Serving an LLM needs memory for three tenants: the weights, the KV cache, and the runtime’s own overhead. Budgeting only for the weights is the classic mistake, and it surfaces as an out-of-memory error the first time someone sends a long prompt. This calculator sizes all three and checks the total against six common GPUs.
What it computes
weights = params × bytes per param / 1,024³
kv/token = 2 × layers × (hidden × kv_heads / heads) × 2 bytes
kv/seq = kv/token × context / 1,024³
kv total = kv/seq × concurrent sequences
overhead = (weights + kv total) × overhead %
total = weights + kv total + overhead
GPUs = ⌈ total / VRAM ⌉ fits when total ≤ VRAM- Bytes per param is 2 for FP16 or BF16, 1 for INT8 and 0.5 for INT4. It applies to the weights only.
- kv_heads / heads is the grouped-query attention ratio. Sharing each key-value head across several query heads shrinks the cache by that factor.
- The KV cache is always sized at 2 bytes per value, and every concurrent sequence is assumed to hold a full-length context.
- Overhead, 15% by default, covers activations, CUDA workspaces and allocator fragmentation.
Sizes are in 1,024-based gigabytes. The full fine-tune card is a separate reference: parameters times 16 bytes.
A worked example
The defaults are Llama 3.1 8B (8.03B parameters, 32 layers, hidden size 4,096, 32 heads, 8 KV heads) in FP16, an 8,192-token context, one sequence and 15% overhead.
- Weights:
8.03 × 10⁹ × 2 = 16.06 billion bytes = 14.96 GB, shown as 15 GB. - KV dimension:
4,096 × 8 / 32 = 1,024, so2 × 32 × 1,024 × 2 = 131,072 bytesper token. - KV cache:
131,072 × 8,192 = 1,073,741,824 bytes, exactly 1 GB, for the single sequence. - Overhead:
(14.96 + 1) × 0.15 = 2.39 GB. - Total:
14.96 + 1 + 2.39 = 18.35, shown as 18.4 GB. It does not fit a 16 GB T4 (2× needed) and fits every card from 24 GB up.
| Change one input | KV cache | Total | Smallest card that fits |
|---|---|---|---|
| Defaults | 1 GB | 18.4 GB | 24 GB |
| INT4 weights | 1 GB | 5.5 GB | 16 GB |
| 128K context | 16 GB | 35.6 GB | 40 GB |
| 32 sequences at 8K | 32 GB | 54 GB | 80 GB |
The last two rows are the point: past a few users or a long context, the cache costs more than the model.
Assumptions and limits
- Every sequence at full context. Real traffic mixes short and long requests, and paged attention allocates cache as tokens arrive, so average use is lower than this worst case.
- FP16 KV cache only. Engines that store the cache in FP8 or INT4 need half or a quarter of the KV figure.
- A flat overhead. Activation memory depends on batch size and engine settings, and some engines reserve a fixed share of VRAM up front.
- Naive multi-GPU counts. Splitting a model across cards adds communication buffers and duplicated pieces, so the count needed can be higher.
- Mixture-of-experts models such as Mixtral need all experts in memory, which is why the preset uses the total of 46.7B parameters, not the active ones.
How to use the result
- Leave headroom. Plan for the total to sit 10 to 20% below the card’s capacity, because drivers and the display or other processes take a share.
- Size for the traffic you promise. Set concurrent sequences to your peak users and context to your longest supported prompt.
- Quantize weights first. INT4 often moves a model down a whole card class; then reduce context or concurrency if the KV cache still dominates.
- Prefer GQA models when long context matters, and check the KV row before buying bigger GPUs.
How it works
- Weights = parameters × bytes per parameter (FP16 = 2, INT8 = 1, INT4 = 0.5).
- KV cache/token = 2 × layers × (hidden × kv-heads ÷ heads) × 2 bytes; GQA shrinks it.
- Total = (weights + KV × concurrent sequences) × (1 + overhead %).
- Architecture presets use the models’ published configs.
Frequently asked questions
How much VRAM does an LLM need?
Weights need the parameter count times the bytes per parameter: an 8B model is about 15 GB in FP16, 7.5 GB in INT8 and 3.7 GB in INT4. On top of that, every in-flight request holds a KV cache that grows with context length, plus runtime overhead for activations, buffers and fragmentation. With this tool’s defaults, Llama 3.1 8B in FP16 at an 8,192-token context for one user needs about 18.4 GB, so it fits a 24 GB card but not a 16 GB one.
What is the KV cache and why does it matter?
The KV cache stores the attention keys and values for every token in the context, so generation does not reprocess the whole prompt for each new token. It grows linearly with context length and with the number of concurrent sequences. For Llama 3.1 8B it is 128 KB per token, so 1 GB at 8K tokens and 16 GB at 128K, more than the 15 GB of FP16 weights. It is usually what limits how many users fit on one GPU.
How does quantization change the requirement?
INT8 halves and INT4 quarters the weight memory compared with FP16, usually with a modest quality cost. In this tool, Llama 3.1 70B at an 8K context needs about 154 GB in FP16, two 80 GB cards, but about 40.7 GB in INT4, which fits one 80 GB card or two 24 GB cards. The tool keeps the KV cache in FP16 regardless, so quantizing weights does not shrink context memory; KV-cache quantization is a separate technique.
How do I calculate KV cache size per token?
Multiply 2, for keys and values, by the number of layers, by the KV dimension, by the bytes per value. The KV dimension is the hidden size times the number of KV heads divided by the number of attention heads. Llama 3.1 8B has 32 layers, a 4,096 hidden size, 32 heads and 8 KV heads, so the KV dimension is 1,024 and the cache is 2 times 32 times 1,024 times 2 bytes, or 131,072 bytes per token at FP16.
Why does fine-tuning need so much more memory than inference?
Full fine-tuning with AdamW holds the weights, the gradients and two optimizer moments for every parameter, roughly 16 bytes per parameter before activations. That is about 120 GB for an 8B model, against about 15 GB to serve it in FP16. That gap is why parameter-efficient methods such as LoRA and QLoRA exist: they freeze the base weights and train small adapters, so the optimizer state stays tiny and the job fits on far less hardware.