Scaling Laws
The empirical, roughly power-law relationship between a model’s size, its training data, its compute budget, and its loss.
Scaling laws describe how a language model’s loss falls predictably as you scale up parameters, data, and compute together — smoothly enough to extrapolate performance of a not-yet-trained larger model. The Chinchilla scaling laws in particular showed many early large models were undertrained relative to their size — for a fixed compute budget, a smaller model trained on more data often beats a larger model trained on less.