Transformer Math
The napkin formulas behind every scaling, memory and cost conversation — params, FLOPs, KV cache, VRAM.
Parameters & compute
- Params (dense)
- ≈
12 · L · d²(L layers, d hidden) — attention 4d², MLP 8d² - Training FLOPs
- ≈
6 · params · tokens(2 forward + 4 backward) - Inference FLOPs
- ≈
2 · paramsper generated token (dense) - Chinchilla-optimal
- ≈
20 tokens per parameter(modern models train far past it) - MoE active params
- compute follows active params; memory follows total
Memory
- Weights
params × bytes— FP16 = 2, INT8 = 1, INT4 = 0.5 B/param- KV cache / token
2 · L · (d · kv_heads / heads) · bytes— GQA divides it- Full fine-tune (AdamW)
- ≈
16 B/param(weights + grads + 2 moments) - LoRA / QLoRA
- ≈
2.5/0.75 B/param— the base dominates - Rule of thumb
- 7B FP16 ≈ 14 GB · 70B FP16 ≈ 140 GB · INT4 quarters it
Throughput & latency
- Decode speed ceiling
- ≈
memory bandwidth ÷ model bytes(bandwidth-bound) - Prefill
- compute-bound, parallel over prompt — thousands of tok/s
- TTFT
- ≈ queue +
prompt_tokens ÷ prefill_rate(+ network) - Speculative decoding
- draft k tokens, verify in 1 pass — 2-3× decode, identical output
- Batching
- raises throughput, not per-user speed — decode stays bandwidth-bound
Cost
- API cost
in_tok/1M × in_price + out_tok/1M × out_price; output ≈ 3-5× input- Blended $/1M
(ratio · in + out) / (ratio + 1)at in:out ratio- History growth
- chat resends history each turn — input tokens grow ~quadratically
- Prompt caching
- cached prefix tokens ~10× cheaper; keep stable parts first