Scaling Laws,
explained.

By 2020, "bigger models are better" was folk wisdom, but folk wisdom is a terrible basis for spending millions on a training run. Nobody could say how much better a model would get if you doubled its size, or whether returns would suddenly flatten. Every large run was, in part, a leap of faith.

The Kaplan et al. paper asked the question as a scientist would: hold everything else fixed and measure exactly how a Transformer's test loss — its next-word prediction error — changes as you vary model size, data, and compute across many orders of magnitude. Not a hunch. A curve.

The headline result is startlingly clean. Plot loss against model size (or data, or compute) on a log-log scale and you get a straight line — the signature of a power law. Over seven orders of magnitude, the relationship holds: each time you multiply the resource, loss drops by a predictable, constant fraction.

Straight lines on a log-log plot are the same math that describes earthquakes and city sizes. Finding one inside neural networks meant that scaling wasn't chaotic or task-specific luck — it was a regular, extrapolatable trend. And crucially, within the range studied there was no wall: the line kept going down.

Loss responds to three resources, each its own power law when it's the bottleneck:

The key insight is the interaction: loss is set by whichever resource is scarcest. A giant model starved of data plateaus; a mountain of data fed to a tiny model plateaus too. To keep loss falling you have to grow all three together — and the paper gives a combined formula predicting loss from N and D at once.

Here's the practical magic. Because the curves are so regular, you can run a handful of small, cheap experiments, fit the power law, and extrapolate to predict the loss of a model 100× or 1000× bigger — one you haven't built yet. The forecast is close enough to plan around.

That converts a gamble into an engineering decision. Instead of "let's train a huge model and pray," a lab can say "our curve predicts this budget yields this loss, which should unlock these capabilities — worth it." Scaling laws made the cost of frontier models justifiable to the people signing the checks.

The sharpest practical question: if you have a fixed pile of compute, should you train a bigger model or train on more data? The Kaplan paper's answer was to lean heavily toward size — make the model much bigger and, perhaps surprisingly, don't train it all the way to convergence; stop early and spend the compute on parameters instead.

This prescription directly shaped the era: GPT-3 is enormous (175B) but trained on comparatively modest data. It was the scaling laws saying "go big." Two years later, Chinchilla revisited the analysis and found a correction — those models were undertrained. For truly compute-optimal training, parameters and tokens should grow in roughly equal measure, meaning smaller models on much more data. The power-law framework survived; the recommended recipe flipped.

Scaling laws are the intellectual license for the entire modern-LLM era. They gave labs the confidence — and the numbers — to commit to nine-figure training runs, because the payoff could be estimated in advance. The chain runs straight from this paper to GPT-3 and every frontier model after it.

They also reframed research priorities. If loss falls predictably with scale, then compute and data are first-class levers, sometimes beating architectural cleverness. A lot of the last few years' progress is, bluntly, this paper's curves being ridden upward.

Scaling laws measure loss, not usefulness. Lower next-word error correlates with capability, but the mapping is loose — a big drop in loss can yield a small or a surprising jump in real skill (emergent abilities). The laws forecast the curve, not exactly which abilities appear where on it.

They also have boundaries: the exponents depend on architecture and data quality, the specific compute-optimal ratio was revised by Chinchilla, and no power law can extend forever — eventually data runs out or the curve must bend. And "just scale" says nothing about alignment, safety, or truthfulness, which don't come for free with size.

This is the paper that turned scaling from a vibe into a discipline. Every time a lab announces a bigger model with a straight face about expected gains, they're invoking scaling laws. The practice of fitting small-scale curves and extrapolating to plan a flagship run is now standard operating procedure across the industry.

More deeply, it changed what the field believes. The "scaling hypothesis" — that a large fraction of intelligence-like behavior falls out of scale applied to a simple objective — got its empirical backbone here. Whether that belief is exactly right is still argued, but it has driven the trajectory of AI more than almost any single idea since the Transformer itself.