Chinchilla,
explained.

This rung exists to overturn the one below it, and it is worth being precise about how narrow the disagreement is. Rung 4 and this paper agree that test loss follows a power law in compute; they agree on the method of fitting curves to smaller runs and extrapolating; they measure the same quantity, the compute-optimal split of a fixed budget between parameters and training tokens, in the same units. They disagree only on the answer — and the disagreement is large enough to have misallocated the field's compute for two years.

Rung 4: raise compute, and grow parameters several times faster than data. This paper: grow them at roughly the same rate, which lands on the rule of thumb of about 20 training tokens per parameter.

Now put the ladder's numbers through that rule. GPT-3: 175B parameters on ~300B tokens — about 1.7 tokens per parameter, roughly a twelfth of optimal. Gopher, the 280B model this paper's authors had trained the year before on the same ~300B tokens, is about 1.1. Both were not merely suboptimal; they were spending most of their budget on the wrong side of the trade.

The proof is the part to take seriously, because it is a like-for-like swap rather than a new record: the same compute budget as Gopher, spent at the predicted ratio, produces a 70B model trained on 1.4T tokens — a quarter of the parameters, about four and a half times the data — and it outperforms the 280B model it replaces. A smaller model that is also cheaper to serve. The exercise below re-derives that 70B from Gopher's budget using nothing but the two rules.

After GPT-3, the headline number was parameter count. 175 billion, then 280 billion (Gopher), then 530 billion. Each lab poured its compute budget into stacking more parameters, on the assumption that size was the lever that mattered most.

But every model is really a bet about how to spend one thing — a fixed compute budget — across two knobs: how many parameters the model has, and how many tokens of data it trains on. The whole field had quietly turned one knob to the maximum and left the other alone.

DeepMind asked it precisely: for a fixed amount of compute, what split between parameters and data gives the best model? To answer, they trained over 400 models across a wide range of sizes and data amounts, and fit curves to how loss falls as you trade one knob against the other.

The answer contradicted the entire industry.

To use compute optimally, parameters and training tokens should grow together — roughly in equal proportion. Their rule of thumb: about 20 training tokens per parameter. A 10-billion-parameter model wants ~200 billion tokens; double the model, double the data.

Two rules turn that into an exact recipe. Training compute is C ≈ 6·N·D (params × tokens), and the optimal split sets D = 20·N. Substitute one into the other and the whole budget collapses to a single unknown:

Give me a compute budget and this returns the one best model size — no sweep, just a square root.

By that measure, GPT-3 and Gopher were wildly out of balance — enormous models fed far too little data. They were undertrained: all that parameter capacity sat half-empty because the models never saw enough text to fill it.

To settle it, they built Chinchilla: same compute budget as their own 280B Gopher, but spent the compute-optimal way — 70 billion parameters (a quarter the size) trained on 1.4 trillion tokens (about four times the data).

Same total compute — just a different split between the two knobs.

Chinchilla rewired how the field spends compute. "Chinchilla-optimal" became standard vocabulary, and the trillion-token training runs behind LLaMA and nearly every capable model since are its direct consequence — smaller, data-rich models tuned for both quality and cheap inference.

It's also a caution against the obvious metric. For years "bigger" was the number everyone chased, and it was the wrong knob to max out. The lesson — balance your budget, don't just crank the one number people brag about — outlasts any single model.

Hoffmann and colleagues trained more than 400 language models, from 70M to over 16B parameters, on 5B to 500B tokens of MassiveText, the corpus behind DeepMind's Gopher. All are decoder-only Transformers trained with a cosine learning-rate schedule whose length matches the run. The paper treats that as essential: a cosine cycle set more than about 25% longer than the actual run leaves the model clearly under-trained at the point where it is measured. Training compute is estimated as C ≈ 6ND. The question, which N and D minimize loss at a fixed C, is then answered three independent ways.

1. Approach 1: fixed model sizes and the training-curve envelope. For each model size, from 70M to over 10B parameters, train four runs whose cosine schedules end at token horizons spanning a factor of 16. Smooth and interpolate each training-loss curve, so every run gives loss as a continuous function of FLOPs spent. At each FLOP count, take the lowest loss reached by any run, the envelope, and record the N and D of the run that reached it. Fitting N_opt ∝ C^^a and D_opt ∝ C^^b to those points gives a ≈ 0.50 and b ≈ 0.50.
2. Approach 2: IsoFLOP profiles. Pick nine fixed budgets from 6 × 10^¹⁸ to 3 × 10^²¹ FLOPs. Within each budget, train a range of model sizes, each for exactly as many tokens as the budget allows, and plot final loss against log N. Every profile is a valley; fit a parabola and take its minimum as the best size for that budget. Power laws through the nine minima give a ≈ 0.49 and b ≈ 0.51.
3. Approach 3: a parametric loss. Model every final loss from approaches 1 and 2 as L(N, D) = E + A/N^^α + B/D^^β. Fit the five constants by minimizing a Huber loss (δ = 10^⁻³) between predicted and observed log loss, using L-BFGS started from a grid of initializations; the Huber loss makes the fit robust to outlying runs. Minimizing that L subject to C = 6ND gives the exponents in closed form: a ≈ 0.46 and b ≈ 0.54.
4. Project to large budgets. With approach 1's fit, the optimum at Gopher's budget of about 5.76 × 10^²³ FLOPs is about 67B parameters on 1.5T tokens; a 175B model would want about 3.7T tokens, and a 280B model about 5.9T. These projections sit near 20 tokens per parameter, which is where the popular rule of thumb comes from. All three approaches put the optimum for Gopher's budget well below its 280B parameters, at roughly 40B to 70B.
5. Verify with one large run. Train Chinchilla at Gopher's compute: 70B parameters on 1.4T tokens. It keeps Gopher's architecture family, with 80 layers, d_model 8,192 and 64 heads of size 128, but uses AdamW instead of Adam, a SentencePiece tokenizer without NFKC normalization, a batch that doubles from 1.5M to 3M tokens partway through, and a slightly reweighted MassiveText mix to cover the extra tokens. The forward and backward passes run in bfloat16, with a float32 copy of the weights kept in the optimizer state.

Approaches 1 and 2 agree closely, while approach 3 leans further toward data. That gap is the part a 2024 replication went after.

• E ≈ 1.69 — the irreducible loss: the entropy of natural text, which no model removes.
• A/N^α, with A ≈ 406.4 and α ≈ 0.34 — the shortfall of a perfectly trained model that has only N parameters.
• B/D^β, with B ≈ 410.7 and β ≈ 0.28 — the shortfall from seeing only D tokens, because the model is not trained to convergence.
• C/6 — substitutes C = 6ND, so the frontier is the point on that budget where the two shortfalls trade off at the margin.

At the extremes: with unlimited data the loss floor is E + A/N^^α, and with an unlimited model it is E + B/D^^β. If α equalled β the split would be exactly even, a = b = 0.5. With the published α larger than β, a ≈ 0.46, so data grows a little faster than size. The 2024 refit by Besiroglu and colleagues found α ≈ 0.35 and β ≈ 0.37, which moves a back to about 0.51, in line with approaches 1 and 2.

The three approaches, and the earlier estimate they overturned, in the same units (N_opt ∝ C^^a, D_opt ∝ C^^b):

Then the verification, at Gopher's compute:

Chinchilla outperformed Gopher, GPT-3 and other far larger models on almost every evaluation reported: 7.6 points better on MMLU, an average of 65.1% against 54.4% on BIG-bench, and better on language modeling, reading comprehension, closed-book question answering and common-sense tasks. A deployment bonus comes free: a 70B model is far cheaper and faster to fine-tune and serve than a 280B one.

The negative results

The cycle-length ablation is the paper's own: cosine schedules set much longer than the run produced clearly worse models, the mechanism behind the earlier misallocation. The external one came in 2024, when Besiroglu, Erdil, Barnett and You at Epoch AI reconstructed approach 3's data points from the paper's plot and refit the same functional form. They reported that the published fit matched the reconstructed data poorly, that its confidence intervals were implausibly narrow for a few hundred training runs, and that their refit gave nearly equal exponents. That undercuts approach 3's estimate while supporting the paper's headline conclusion.

The paper's own caveats are load-bearing. The fit is over a specific family of models, tokenizer and training setup, and the authors do not claim the ~20:1 coefficient transfers outside it. They note their largest verification run is a single model, so the confirmation of the extrapolation rests on one data point at the scale that matters most. And they are clear that the objective being optimised is training compute for a target loss — not the total cost of the model over its life.

That last one turned out to be the important limitation, and it was the field rather than the paper that drew the conclusion: if a model is going to serve a great many requests, inference cost dominates, and you should deliberately train past the compute-optimal point — a smaller model on far more tokens than 20:1 — because you pay the extra training once and save on every query. Most widely deployed open-weight models since are trained well beyond Chinchilla-optimal for exactly this reason, which means the rule is routinely and knowingly violated in production. It is not a refutation; it is a different objective function.

How it held up otherwise: the correction itself held. The claim that parameters and data should scale together is now standard, and "Chinchilla-optimal" entered the vocabulary. The trillion-token training runs behind essentially every capable model since are downstream of this page. Rung 6 then attacks the axis this paper does not measure at all.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →