Chinchilla,
explained.

After GPT-3, the headline number was parameter count. 175 billion, then 280 billion (Gopher), then 530 billion. Each lab poured its compute budget into stacking more parameters, on the assumption that size was the lever that mattered most.

But every model is really a bet about how to spend one thing — a fixed compute budget — across two knobs: how many parameters the model has, and how many tokens of data it trains on. The whole field had quietly turned one knob to the maximum and left the other alone.

DeepMind asked it precisely: for a fixed amount of compute, what split between parameters and data gives the best model? To answer, they trained over 400 models across a wide range of sizes and data amounts, and fit curves to how loss falls as you trade one knob against the other.

The answer contradicted the entire industry.

To use compute optimally, parameters and training tokens should grow together — roughly in equal proportion. Their rule of thumb: about 20 training tokens per parameter. A 10-billion-parameter model wants ~200 billion tokens; double the model, double the data.

By that measure, GPT-3 and Gopher were wildly out of balance — enormous models fed far too little data. They were undertrained: all that parameter capacity sat half-empty because the models never saw enough text to fill it.

To settle it, they built Chinchilla: same compute budget as their own 280B Gopher, but spent the compute-optimal way — 70 billion parameters (a quarter the size) trained on 1.4 trillion tokens (about four times the data).

Same total compute — just a different split between the two knobs.

Chinchilla outperformed Gopher, GPT-3, and other far larger models across a broad sweep of benchmarks — language understanding, reasoning, reading comprehension — despite being a fraction of their size.

And a deployment bonus falls out for free: a 70B model is far cheaper and faster to run than a 280B one — which matters enormously once millions of people are querying it every day.

Chinchilla rewired how the field spends compute. "Chinchilla-optimal" became standard vocabulary, and the trillion-token training runs behind LLaMA and nearly every capable model since are its direct consequence — smaller, data-rich models tuned for both quality and cheap inference.

It's also a caution against the obvious metric. For years "bigger" was the number everyone chased, and it was the wrong knob to max out. The lesson — balance your budget, don't just crank the one number people brag about — outlasts any single model.