GPU VRAM Calculator
Pick a model (Llama, Mistral, Qwen, Phi, Mixtral or custom architecture), a precision (FP16 / INT8 / INT4), a context length and batch size — and get the real VRAM requirement: weights, KV cache (GQA-aware), runtime overhead, and a fits-or-not verdict across common GPUs from a T4 to an H200.
The three things eating your VRAM
Serving an LLM needs memory for three distinct tenants. Weights: every parameter stored at your chosen precision — the fixed cost of having the model loaded at all. KV cache: the attention keys and values for every token of every in-flight request — the variable cost that grows with context and concurrency. Overhead: activations, CUDA workspaces and allocator fragmentation, typically 10–20% on top. The classic mistake is budgeting only for weights and discovering at 3am that long contexts OOM the box.
Weights: precision is a 4× lever
FP16 spends 2 bytes per parameter — ~15 GB for an 8B model, ~140 GB for a 70B. INT8 halves that and INT4 quarters it, with a modest quality cost that modern quantization methods keep small. That single knob decides hardware class: a 70B in FP16 wants two 80 GB datacenter GPUs; in INT4 it squeezes onto two consumer 24 GB cards.
KV cache: the sneaky one
Each generated token attends over everything before it, so the server caches keys and values per token, per layer: 2 × layers × kv-dim × 2 bytes per token. Grouped-query attention (GQA) is why modern models stay serviceable — sharing KV heads across query heads cuts the cache 4–8×. Even so, it scales linearly with context and batch: one Llama-70B sequence at 128K context holds ~5 GB of KV — and twenty concurrent users hold twenty caches. That, not compute, is usually what caps a GPU's concurrency.
Training is a different sport
Inference holds weights; full fine-tuning with AdamW holds weights plus gradients plus two optimizer moments — roughly 16 bytes per parameter before activations. An 8B model that serves happily on one 24 GB card wants ~128 GB to fine-tune fully. That gap is the entire reason LoRA exists: freeze the base, train a few million adapter parameters, keep the optimizer state tiny.
How it works
- Weights = parameters × bytes per parameter (FP16 = 2, INT8 = 1, INT4 = 0.5).
- KV cache/token = 2 × layers × (hidden × kv-heads ÷ heads) × 2 bytes — GQA shrinks it.
- Total = weights + KV × concurrent sequences + runtime overhead.
- Architecture presets use the models’ published configs.
Frequently asked questions
How much VRAM does an LLM need?
Weights need parameters × bytes-per-parameter: an 8B model is ~15 GB in FP16, ~7.5 GB in INT8, ~3.7 GB in INT4. On top of that, every in-flight request holds a KV cache that grows with context length, plus 10–20% runtime overhead for activations and buffers. The calculator adds all three.
What is the KV cache and why does it matter?
The KV cache stores the attention keys and values for every token in the context so generation doesn't re-process the whole prompt per token. It grows linearly with context length and batch size — at long contexts it can rival the weights themselves, and it's usually what actually limits how many users fit on one GPU.
How does quantization change the requirement?
INT8 halves and INT4 quarters the weight memory versus FP16, with a small quality cost — which is why a 70B model that needs ~140 GB in FP16 can run on two 24 GB cards in INT4. Note the KV cache is typically kept in FP16 regardless, so quantization shrinks weights, not context memory.
Why does fine-tuning need so much more memory than inference?
Full fine-tuning with AdamW holds weights, gradients and two optimizer moments — roughly 16 bytes per parameter before activations, so ~128 GB for an 8B model. That is exactly why parameter-efficient methods like LoRA exist: freeze the base weights and train tiny adapters instead.