Transformer Math

The napkin formulas behind every scaling, memory and cost conversation — params, FLOPs, KV cache, VRAM.

A transformer’s cost follows from a few shape numbers: parameters N, layers L, hidden size d, context length and bytes per value. From those, napkin formulas give compute, memory and speed to within a factor good enough for planning. Two facts organize everything below. Compute scales with parameters × tokens: about 6N FLOPs per token to train and 2N to generate. And generating one token at a time is limited by memory bandwidth, not compute, because every new token rereads all the weights.

Figures are approximate on purpose. Use them to compare options, then measure.

Parameters & compute

Params (dense): ≈ 12 · L · d² (L layers, d hidden) — attention 4d², MLP 8d²
Training FLOPs: ≈ 6 · params · tokens (2 forward + 4 backward)
Inference FLOPs: ≈ 2 · params per generated token (dense)
Chinchilla-optimal: ≈ 20 tokens per parameter (modern models train far past it)
Compute-optimal scaling: params and tokens both grow with ≈ √compute: 4× compute → 2× model, 2× data
MoE active params: compute follows active params; memory follows total
Embedding params: vocab × d, doubled when input and output embeddings are untied — about 1B in Llama 3 8B
Attention FLOPs: add ≈ 2 · L · n_ctx · d per token (Kaplan et al.) — small next to 2N until context is long

Architecture shapes

d_head = d / n_heads: usually 128; attention scores are scaled by 1/√d_head
FFN width: classic d_ff ≈ 4d; SwiGLU uses 3 matrices with d_ff ≈ 8d/3 to keep ≈ 8d² params
MHA · GQA · MQA: K/V heads = query heads · a few groups (often 8) · one — fewer K/V heads, smaller cache
Llama 3 8B: 32 layers, d = 4096, 32 query and 8 KV heads, d_ff = 14336, 128K vocabulary
Llama 3 70B: 80 layers, d = 8192, 64 query and 8 KV heads, d_ff = 28672

Memory

Weights: params × bytes — FP16 = 2, INT8 = 1, INT4 = 0.5 B/param
Rule of thumb: 7B FP16 ≈ 14 GB · 70B FP16 ≈ 140 GB · INT4 quarters it
KV cache / token: 2 · L · (d · kv_heads / heads) · bytes — GQA divides it
KV cache, worked: Llama 3 8B at FP16: 2·32·8·128·2 B = 128 KiB per token, 1 GiB at 8K context
KV cache × batch: 32 users at 8K on Llama 3 8B ≈ 32 GiB — twice the weights
Full fine-tune (AdamW): ≈ 16 B/param: 16-bit weights and grads, plus FP32 master copy and two moments
LoRA / QLoRA: ≈ 2.5 / 0.75 B/param — the base dominates
Activations: grow with batch × sequence; checkpointing trades ~33% more compute for far less memory
ZeRO-3 / FSDP: shard weights, gradients and optimizer state over N GPUs: ≈ 16 B/param ÷ N each

Throughput & latency

Decode speed ceiling: ≈ memory bandwidth ÷ model bytes (bandwidth-bound)
Prefill: compute-bound, parallel over prompt — thousands of tok/s
TTFT: ≈ queue + prompt_tokens ÷ prefill_rate (+ network)
TPOT: time per output token ≈ model bytes ÷ bandwidth; total latency = TTFT + TPOT × tokens
Speculative decoding: draft k tokens, verify in 1 pass — 2-3× decode, identical output
Batching: raises throughput, not per-user speed — decode stays bandwidth-bound
Compute-bound batch: decode becomes compute-bound near batch ≈ FLOPs ÷ bandwidth, roughly 300 on an H100
Tensor parallelism: split every layer across GPUs in one node; needs NVLink-class bandwidth
Pipeline parallelism: split layers into stages across GPUs or nodes; costs idle “bubble” time

Tokens & text

1 token ≈ 4 characters: about ¾ of an English word; code and non-English text take more tokens
1,000 tokens: ≈ 750 English words, roughly a page and a half of prose
128K context: ≈ 96,000 words — a long novel in one prompt
Perplexity: exp(loss) with loss in nats — loss 2.0 ≈ perplexity 7.4
Temperature: logits ÷ T before softmax: T below 1 sharpens, above 1 flattens; T → 0 is greedy

Training at scale

GPU-hours: 6·N·D ÷ (peak FLOPs × MFU) ÷ 3600; MFU of 30–50% is good
With recomputation: ≈ 8 · N · D — the extra forward pass pays for checkpointed activations
Llama 3 8B data: 15T+ tokens ≈ 1,900 tokens per parameter, about 90× the Chinchilla ratio
Llama 3.1 405B: ≈ 3.8 × 10²⁵ training FLOPs; Meta reports about 30.8M H100 GPU-hours

Cost

API cost: in_tok/1M × in_price + out_tok/1M × out_price; output usually 4–8× the input price
Blended $/1M: (ratio · in + out) / (ratio + 1) at in:out ratio
History growth: chat resends history each turn — input tokens grow ~quadratically
Prompt caching: cached input bills at a steep discount (0.1× on Anthropic’s API); keep stable parts first
Self-host $/1M tokens: $/hr ÷ (tok/s × 3600) × 1e6 — $2/hr at 1,000 tok/s ≈ $0.56
Batch API: non-urgent requests at half price on the major APIs, results within 24 hours
Reasoning tokens: thinking is billed as output — a short answer can carry thousands of hidden tokens

Recipes

Will a 70B model fit, and how fast will it generate?

Weights (FP16):   70e9 × 2 B             = 140 GB  → two 80 GB GPUs
Weights (INT4):   70e9 × 0.5 B           = 35 GB   → one 48–80 GB GPU
KV cache (FP16):  2 × 80 × 1,024 × 2 B   = 320 KiB per token
                  × 8,192 tokens         ≈ 2.5 GiB per sequence
Decode ceiling:   3.35 TB/s ÷ 35 GB      ≈ 95 tokens/s (INT4, one H100)

Bandwidth divided by model bytes is a ceiling: real systems land below it, and reading the KV cache lowers it further as the context grows.

Training compute and cost for a 7B model

Tokens (Chinchilla):  20 × 7e9              = 140e9 tokens
FLOPs:                6 × 7e9 × 140e9        = 5.9e21
Effective H100:       989e12 × 0.40 MFU      = 4.0e14 FLOP/s
GPU-seconds:          5.9e21 ÷ 4.0e14        = 1.5e7 s ≈ 4,100 GPU-hours
Cost at ~$2.50/hr:                           ≈ $10,000

Production models train far past 20 tokens per parameter (Llama 3 8B saw about 1,900), which multiplies this figure by roughly 100.

API cost of one chat conversation

Conversation:    8 turns, each adding 1,500 input and 400 output tokens
Input per turn:  history is resent: 1.5K, 3.4K, 5.3K … 14.8K ≈ 65K in total
Output:          8 × 400 = 3,200 tokens
Example price:   $3 per 1M input, $15 per 1M output
Cost:            65K × $3/1M + 3.2K × $15/1M ≈ $0.20 + $0.05 ≈ $0.24
With caching:    reads at 0.1×, writes at 1.25× the input price ≈ $0.12

Resending history makes input grow with the square of the turn count, which is why long conversations dominate the bill and why caching the shared prefix halves it here.

Self-hosting versus an API

GPU:               1× H100 at ~$2.50/hr ≈ $1,825 per month
Throughput:        assume 2,000 output tokens/s across a busy batch (measure yours)
Monthly capacity:  2,000 × 3,600 × 730 ≈ 5.3B tokens
Cost per 1M:       $1,825 ÷ 5,256 ≈ $0.35, only while the GPU stays busy
At 10% load:       ≈ $3.50 per 1M

Utilization decides the answer: a GPU that sits idle 90% of the time costs ten times more per token, which is usually when a per-token API wins.

Questions about Transformer Math Cheat Sheet

How much GPU memory do I need to run an LLM?

Start with the weights: parameters times bytes per parameter, so 2 bytes at FP16 or BF16, 1 byte at INT8 and about half a byte at 4-bit. A 7B model needs roughly 14 GB at FP16 and 4 to 5 GB at 4-bit, while a 70B model needs about 140 GB at FP16 and 35 to 40 GB at 4-bit. Then add the KV cache, which grows with context length and with the number of concurrent requests and can exceed the weights at long contexts, plus 10 to 20 percent for activations and framework overhead.

Why is LLM inference memory-bandwidth bound?

Generating each new token requires reading every weight of the model from GPU memory, but the arithmetic per weight is tiny: about two floating-point operations per parameter for one sequence. A modern GPU can do hundreds of operations in the time it takes to fetch one byte, so during decoding the compute units mostly wait for memory. That is why tokens per second track memory bandwidth divided by model size, why quantization speeds up decoding, and why batching raises throughput: each weight fetched from memory then serves many sequences.

What does Chinchilla optimal mean?

It refers to DeepMind’s 2022 Chinchilla study, which found that for a fixed training compute budget, loss is minimized by scaling model size and training tokens roughly equally, at about 20 tokens per parameter. Many earlier models had been too large for the data they saw. Today’s models deliberately train far beyond that point, often a hundred times more tokens per parameter, because a smaller model trained longer costs more to train but much less to serve, and serving costs dominate once a model has millions of users.

How do you calculate the KV cache size?

For each token, every layer stores one key vector and one value vector per key-value head. Multiply 2 for keys and values by the number of layers, the number of key-value heads, the head dimension and the bytes per value. Llama 3 8B has 32 layers and 8 key-value heads of dimension 128, so at FP16 that is 2 times 32 times 8 times 128 times 2 bytes, or 128 KiB per token. That comes to about 1 GiB for an 8K context, multiplied again by the number of concurrent sequences.

Go deeper

GPU VRAM Calculator →KV Cache Calculator →Fine-Tuning Cost Estimator →Transformers handbook →Napkin math cheatsheet →