Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path
Training Compute Calculator
Put in a model size and a token budget and see the training compute it implies — the classic 6ND FLOPs, the GPU-hours and wall-clock time on your cluster, the dollar cost, and how your token budget compares to the Chinchilla compute-optimal 20× rule. Every number is editable, from GPU peak FLOP/s to model-FLOPs-utilization.
Training FLOPs use the 6ND approximation (≈ 6 FLOPs per parameter per token). Wall-clock divides by GPUs × peak × MFU. A first-order planning estimate — real runs pay extra for restarts, checkpointing, warmup and data stalls.
Every large model starts as a compute budget. Before a GPU spins up, teams choose the model size and the number of training tokens, and those two numbers multiplied by a small constant fix the FLOPs, the GPU-hours, the wall-clock time and the bill. This calculator walks that chain and checks your token budget against the Chinchilla rule.
What it computes
FLOPs = 6 × N × D N params, D tokens
throughput = GPUs × peak TFLOP/s × 10¹² × MFU
seconds = FLOPs / throughput
GPU-hours = seconds × GPUs / 3,600
cost = GPU-hours × price per GPU-hour
petaFLOP-days = FLOPs / 10¹⁵ / 86,400
tokens/param = D / N Chinchilla target ≈ 20- Peak is the GPU’s dense BF16 rate. Presets are A100 312, H100 990, H200 990, B200 2,250, MI300X 1,300 and TPU v5e 197 TFLOP/s, each with an illustrative on-demand price, and both are editable.
- MFU is the share of peak you sustain; 40% is the default.
- The Chinchilla banner reads on target within 15% of 20 tokens per parameter, over-trained above that and under-trained below.
A worked example
The defaults are 7B parameters, 2,000B tokens, 1,024 H100s at 990 TFLOP/s, 40% MFU and $3.00 per GPU-hour.
- FLOPs:
6 × 7×10⁹ × 2×10¹² = 8.40 × 10²², about 972 petaFLOP-days. - Throughput:
1,024 × 990×10¹² × 0.40 = 4.05 × 10¹⁷FLOP/s. - Time:
8.4×10²² / 4.05×10¹⁷ = 207,000 s, shown as 2.4 days. - GPU-hours:
207,000 × 1,024 / 3,600 = 58,923. Cost:58,923 × $3 = $176,768, shown as $176.8K. - Tokens per parameter:
2,000 / 7 = 286, which is 14.3× over the 140B-token optimum, a deliberately over-trained model.
| Scenario | FLOPs | Wall-clock | Cost |
|---|---|---|---|
| 7B, 2,000B tokens, H100 ×1,024 | 8.40 × 10²² | 2.4 days | $176.8K |
| 7B, 140B tokens (20×) | 5.88 × 10²¹ | 4.0 hr | $12.4K |
| 70B, 1,400B tokens (20×) | 5.88 × 10²³ | 16.8 days | $1.24M |
| 7B, 2,000B tokens, A100 ×1,024 at $1.80 | 8.40 × 10²² | 7.6 days | $336.5K |
The last row shows cost is not just speed: an A100 is about three times slower, and even at 60% of the hourly price the run costs nearly twice as much.
Assumptions and limits
- Dense transformer. Mixture-of-experts models should enter their active parameter count, and non-transformer architectures follow different rules.
- Perfect scaling. One MFU number covers communication, bubbles and stalls, and real utilization falls as clusters grow.
- No overhead. Failures, restarts, evaluation, ablations and failed experiments are excluded and usually exceed the final run.
- Illustrative prices. GPU rates vary by provider, commitment and region, and the presets are not quotes.
- Pre-training only. Post-training, fine-tuning and inference are separate budgets.
How to use the result
- Start from the budget. Divide your FLOPs by what the cluster delivers to see whether a model size is feasible in the time you have.
- Choose tokens per parameter on purpose. Near 20 for the best quality per training FLOP; far higher if the model will serve heavy traffic and cheaper inference matters.
- Run an MFU sensitivity. Compare 30%, 40% and 50%; the spread is a good estimate of schedule risk.
- Multiply the estimate by two or three to cover experiments and reruns, since the final run is usually a minority of what a project actually spends.
- Mind inference too. An over-trained small model costs more to train but can be cheaper to serve every day, so compare lifetime cost, not training cost alone.
How it works
- Training FLOPs ≈ 6 × N (params) × D (tokens).
- Effective throughput = GPUs × peak FLOP/s × MFU.
- Wall-clock = total FLOPs ÷ effective throughput; cost = GPU-hours × price.
- Chinchilla compute-optimal ≈ 20 tokens per parameter.
Frequently asked questions
Where does the 6ND formula come from?
Training a dense transformer costs roughly 6 FLOPs per parameter per token: about 2 for the forward pass, a multiply and an add per weight, and about 4 for the backward pass, which computes gradients with respect to both activations and weights. Total training FLOPs are therefore about 6 times the parameter count N times the token count D. It ignores attention’s quadratic term and embedding overhead, which are small at typical sequence lengths, but it is the number practitioners plan with.
What is Chinchilla-optimal?
DeepMind’s Chinchilla study found that, for a fixed compute budget, loss is lowest when parameters and training tokens grow together at roughly 20 tokens per parameter, far more data than earlier models used. A compute-optimal 70B model therefore wants about 1.4 trillion tokens. Training beyond 20 tokens per parameter still improves the model; it is just not the most loss per FLOP, which is why models meant for heavy inference use are deliberately over-trained.
What is MFU and why is it not 100%?
Model FLOPs utilization is the fraction of a GPU’s peak FLOP/s that training actually sustains, after memory bandwidth limits, communication between GPUs, pipeline bubbles and kernel gaps. Well-tuned large-scale training commonly lands around 35 to 50%, and the calculator defaults to 40%. Peak FLOP/s times MFU is the effective throughput that the total FLOPs are divided by, so a 10-point change in MFU moves time and cost by about a quarter.
How much does it cost to train a 7B model?
With the defaults, a 7B model on 2 trillion tokens needs 8.4 × 10²² FLOPs. On 1,024 H100s at 990 TFLOP/s, 40% MFU and $3 per GPU-hour, that is about 2.4 days, 58,900 GPU-hours and roughly $177K. At the Chinchilla-optimal 140 billion tokens it falls to about 4 hours and $12K. Prices and utilization vary widely, so treat these as the tool’s illustrative defaults and substitute your own rates.
Why is my estimate different from a real training run?
This is a first-order planning estimate. Real runs differ because of restarts and hardware failures, evaluation and checkpointing overhead, data-loading stalls, a warmup where utilization is low, and architecture-specific costs: mixture-of-experts models activate only some parameters, and long context inflates attention. Use the calculator to size the order of magnitude and compare scenarios, not to predict a cloud invoice to the dollar.