Training Compute Calculator
Put in a model size and a token budget and see the training compute it implies — the classic 6ND FLOPs, the GPU-hours and wall-clock time on your cluster, the dollar cost, and how your token budget compares to the Chinchilla compute-optimal 20× rule. Every number is editable, from GPU peak FLOP/s to model-FLOPs-utilization.
Training FLOPs use the 6ND approximation (≈ 6 FLOPs per parameter per token). Wall-clock divides by GPUs × peak × MFU. A first-order planning estimate — real runs pay extra for restarts, checkpointing, warmup and data stalls.
Every large model starts as a compute budget. Before a single GPU spins up, teams decide how big the model will be and how many tokens it will see — and those two numbers, multiplied by a small constant, fix almost everything else: the FLOPs, the GPU-hours, the wall-clock, and the bill.
The constant is 6. A dense transformer spends about two FLOPs per parameter per token on the forward pass and about four on the backward pass, so training cost is 6 × N × D — parameters times tokens times six. A 7B model on 2T tokens is 6 × 7e9 × 2e12 ≈ 8.4e22 FLOPs. Divide that by what your cluster actually delivers — GPU count times peak FLOP/s times a realistic MFU of 35–50% — and you have the hours. Multiply the hours by your GPU price and you have the cost.
The last dial is the token budget itself. Chinchilla showed that, per unit of compute, loss falls fastest when you feed roughly 20 tokens per parameter — so a compute-optimal 70B model wants about 1.4T tokens, not the few-hundred-billion earlier models used. Training beyond 20× is not wasted; it trades training compute for a cheaper, stronger model at inference time, which is why so many shipping models are deliberately over-trained. This calculator makes that trade visible: change the tokens, watch the compute and the Chinchilla ratio move together.
How it works
- Training FLOPs ≈ 6 × N (params) × D (tokens).
- Effective throughput = GPUs × peak FLOP/s × MFU.
- Wall-clock = total FLOPs ÷ effective throughput.
- Chinchilla compute-optimal ≈ 20 tokens per parameter.
Frequently asked questions
Where does the 6ND formula come from?
Training a dense transformer costs roughly 6 FLOPs per parameter per token: about 2 for the forward pass (a multiply and an add per weight) and about 4 for the backward pass, which computes gradients with respect to both activations and weights. So total training FLOPs ≈ 6 × N params × D tokens. It is an approximation — it ignores attention’s quadratic term (small at typical sequence lengths relative to the MLPs) and embedding/normalization overhead — but it is the number practitioners plan with.
What is Chinchilla-optimal?
DeepMind’s Chinchilla paper found that, for a fixed compute budget, loss is minimised when tokens and parameters scale together at roughly 20 tokens per parameter — far more data than earlier models used. So a compute-optimal 70B model wants about 1.4T training tokens. Training past 20× still helps (it just is not the most loss-per-FLOP), which is why inference-focused models are deliberately “over-trained.”
What is MFU and why is it not 100%?
Model FLOPs Utilization is the fraction of a GPU’s peak FLOP/s your training actually sustains, after memory bandwidth limits, communication between GPUs, pipeline bubbles and kernel gaps. Well-tuned large-scale training lands around 35–50%; the calculator defaults to 40%. Peak FLOP/s times MFU is the effective throughput you divide the total FLOPs by.
Why is my estimate different from a real training run?
This is a first-order planning estimate. Real runs differ because of restarts and failures, evaluation and checkpointing overhead, data-loading stalls, a warmup where MFU is low, and architecture-specific costs (mixture-of-experts changes the active-parameter count, long context inflates attention). Use it to size the order of magnitude and compare scenarios, not to bill a cloud invoice.