Chinchilla,
explained.

This rung exists to overturn the one below it, and it is worth being precise about how narrow the disagreement is. Rung 4 and this paper agree that test loss follows a power law in compute; they agree on the method of fitting curves to smaller runs and extrapolating; they measure the same quantity, the compute-optimal split of a fixed budget between parameters and training tokens, in the same units. They disagree only on the answer — and the disagreement is large enough to have misallocated the field's compute for two years.

Rung 4: raise compute, and grow parameters several times faster than data. This paper: grow them at roughly the same rate, which lands on the rule of thumb of about 20 training tokens per parameter.

Now put the ladder's numbers through that rule. GPT-3: 175B parameters on ~300B tokens — about 1.7 tokens per parameter, roughly a twelfth of optimal. Gopher, the 280B model this paper's authors had trained the year before on the same ~300B tokens, is about 1.1. Both were not merely suboptimal; they were spending most of their budget on the wrong side of the trade.

The proof is the part to take seriously, because it is a like-for-like swap rather than a new record: the same compute budget as Gopher, spent at the predicted ratio, produces a 70B model trained on 1.4T tokens — a quarter of the parameters, about four and a half times the data — and it outperforms the 280B model it replaces. A smaller model that is also cheaper to serve. The exercise below re-derives that 70B from Gopher's budget using nothing but the two rules.

After GPT-3, the headline number was parameter count. 175 billion, then 280 billion (Gopher), then 530 billion. Each lab poured its compute budget into stacking more parameters, on the assumption that size was the lever that mattered most.

But every model is really a bet about how to spend one thing — a fixed compute budget — across two knobs: how many parameters the model has, and how many tokens of data it trains on. The whole field had quietly turned one knob to the maximum and left the other alone.

DeepMind asked it precisely: for a fixed amount of compute, what split between parameters and data gives the best model? To answer, they trained over 400 models across a wide range of sizes and data amounts, and fit curves to how loss falls as you trade one knob against the other.

The answer contradicted the entire industry.

To use compute optimally, parameters and training tokens should grow together — roughly in equal proportion. Their rule of thumb: about 20 training tokens per parameter. A 10-billion-parameter model wants ~200 billion tokens; double the model, double the data.

Two rules turn that into an exact recipe. Training compute is C ≈ 6·N·D (params × tokens), and the optimal split sets D = 20·N. Substitute one into the other and the whole budget collapses to a single unknown:

Give me a compute budget and this returns the one best model size — no sweep, just a square root.

By that measure, GPT-3 and Gopher were wildly out of balance — enormous models fed far too little data. They were undertrained: all that parameter capacity sat half-empty because the models never saw enough text to fill it.

To settle it, they built Chinchilla: same compute budget as their own 280B Gopher, but spent the compute-optimal way — 70 billion parameters (a quarter the size) trained on 1.4 trillion tokens (about four times the data).

Same total compute — just a different split between the two knobs.

Chinchilla outperformed Gopher, GPT-3, and other far larger models across a broad sweep of benchmarks — language understanding, reasoning, reading comprehension — despite being a fraction of their size.

And a deployment bonus falls out for free: a 70B model is far cheaper and faster to run than a 280B one — which matters enormously once millions of people are querying it every day.

Chinchilla rewired how the field spends compute. "Chinchilla-optimal" became standard vocabulary, and the trillion-token training runs behind LLaMA and nearly every capable model since are its direct consequence — smaller, data-rich models tuned for both quality and cheap inference.

It's also a caution against the obvious metric. For years "bigger" was the number everyone chased, and it was the wrong knob to max out. The lesson — balance your budget, don't just crank the one number people brag about — outlasts any single model.

The paper's own caveats are load-bearing. The fit is over a specific family of models, tokenizer and training setup, and the authors do not claim the ~20:1 coefficient transfers outside it. They note their largest verification run is a single model, so the confirmation of the extrapolation rests on one data point at the scale that matters most. And they are clear that the objective being optimised is training compute for a target loss — not the total cost of the model over its life.

That last one turned out to be the important limitation, and it was the field rather than the paper that drew the conclusion: if a model is going to serve a great many requests, inference cost dominates, and you should deliberately train past the compute-optimal point — a smaller model on far more tokens than 20:1 — because you pay the extra training once and save on every query. Most widely deployed open-weight models since are trained well beyond Chinchilla-optimal for exactly this reason, which means the rule is routinely and knowingly violated in production. It is not a refutation; it is a different objective function.

How it held up otherwise: the correction itself held. The claim that parameters and data should scale together is now standard, and "Chinchilla-optimal" entered the vocabulary. The trillion-token training runs behind essentially every capable model since are downstream of this page. Rung 6 then attacks the axis this paper does not measure at all.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →