Scaling Laws
The empirical, roughly power-law relationship between a model’s size, its training data, its compute budget, and its loss.
Scaling laws describe how a language model’s loss falls predictably as you scale up parameters, data, and compute together — smoothly enough to extrapolate performance of a not-yet-trained larger model. The Chinchilla scaling laws in particular showed many early large models were undertrained relative to their size — for a fixed compute budget, a smaller model trained on more data often beats a larger model trained on less.
Worked example: scaling laws say quality improves predictably with parameters, data, and compute — and Chinchilla found a compute-optimal balance of ~20 training tokens per parameter, so a 10B model wants ~200B tokens. Gotcha: the laws describe smooth loss curves, not capability jumps, and they have limits (data runs out, some abilities emerge non-smoothly); ‘just scale it’ is a real lever, not the whole story — data quality and architecture matter too.