Scaling Laws,
explained.

Rung 3 built one very large model and reported what it could do. This rung asks the question that makes such a run repeatable: given a compute budget, how large should the model be and how much data should it see? The answer is empirical and, for the first time, predictive — test loss falls as a smooth power law in parameters, in data and in compute over several orders of magnitude, which means you can fit the curve on small runs and extrapolate to the run you cannot afford to repeat.

That is a genuine change in kind. Rung 3's sizing was a consequence of the hardware; rung 4's is a derivation. It is also the origin of the practice of deciding a model's shape before training it rather than after.

And then the specific answer, in the units this ladder has been carrying: when compute grows, the paper concludes that most of it should go into parameters and comparatively little into data — as a rule of thumb, model size should grow several times faster than the dataset. Under that allocation GPT-3's 175B-on-300B does not look like a mistake; it looks close to correct. Read rung 5 next, and read it as a direct reply: it measures the same quantity, in the same units, and gets the opposite allocation.

By 2020, "bigger models are better" was folk wisdom, but folk wisdom is a terrible basis for spending millions on a training run. Nobody could say how much better a model would get if you doubled its size, or whether returns would suddenly flatten. Every large run was, in part, a leap of faith.

The Kaplan et al. paper asked the question as a scientist would: hold everything else fixed and measure exactly how a Transformer's test loss — its next-word prediction error — changes as you vary model size, data, and compute across many orders of magnitude. Not a hunch. A curve.

The headline result is startlingly clean. Plot loss against model size (or data, or compute) on a log-log scale and you get a straight line — the signature of a power law. Over seven orders of magnitude, the relationship holds: each time you multiply the resource, loss drops by a predictable, constant fraction.

Straight lines on a log-log plot are the same math that describes earthquakes and city sizes. Finding one inside neural networks meant that scaling wasn't chaotic or task-specific luck — it was a regular, extrapolatable trend. And crucially, within the range studied there was no wall: the line kept going down.

Loss responds to three resources, each its own power law when it's the bottleneck:

The key insight is the interaction: loss is set by whichever resource is scarcest. A giant model starved of data plateaus; a mountain of data fed to a tiny model plateaus too. To keep loss falling you have to grow all three together — and the paper gives a combined formula predicting loss from N and D at once.

The paper's three fitted power laws, with their actual exponents (N = non-embedding parameters, D = tokens, C = compute):

Tiny exponents — and that's the whole story of the modern AI economy. A power law with exponent 0.076 means improvement never stops, but gets brutally expensive. Put numbers on it:

Read that last row again: two orders of magnitude more money buys one-fifth off the loss. This is why "just scale it" is simultaneously true and terrifying — the curve never saturates in the measured range (the paper verified it across seven orders of magnitude of compute), but every step up it costs 10× the last one.

The Kaplan allocation rule falls out of balancing the laws: with 10× more compute, spend it as ~5.4× more parameters but only ~1.9× more data (N ∝ C^^0.73, D ∝ C^^0.27). That "parameters first" split is exactly the prescription Chinchilla later corrected — the laws survived, the constants didn't.

Here's the practical magic. Because the curves are so regular, you can run a handful of small, cheap experiments, fit the power law, and extrapolate to predict the loss of a model 100× or 1000× bigger — one you haven't built yet. The forecast is close enough to plan around.

That converts a gamble into an engineering decision. Instead of "let's train a huge model and pray," a lab can say "our curve predicts this budget yields this loss, which should unlock these capabilities — worth it." Scaling laws made the cost of frontier models justifiable to the people signing the checks.

The sharpest practical question: if you have a fixed pile of compute, should you train a bigger model or train on more data? The Kaplan paper's answer was to lean heavily toward size — make the model much bigger and, perhaps surprisingly, don't train it all the way to convergence; stop early and spend the compute on parameters instead.

This prescription directly shaped the era: GPT-3 is enormous (175B) but trained on comparatively modest data. It was the scaling laws saying "go big." Two years later, Chinchilla revisited the analysis and found a correction — those models were undertrained. For truly compute-optimal training, parameters and tokens should grow in roughly equal measure, meaning smaller models on much more data. The power-law framework survived; the recommended recipe flipped.

The paper's contribution is a measurement procedure: hold the model family and data fixed, vary one resource at a time over orders of magnitude, and fit curves that can be combined. In order:

1. Fix one model family and one dataset. Decoder-only Transformers trained on WebText2, an extended WebText of 20.3M documents and 96 GB of text, about 2.3 × 10^¹⁰ BPE tokens with a 50,257-token vocabulary. Another 6.6 × 10^⁸ tokens are held out to measure test loss. The context is 1,024 tokens.
2. Count parameters the useful way. N excludes embeddings: N ≈ 2·n_layer·d_model·(2·d_attn + d_ff), which is 12·n_layer·d_model^² for the standard shape. Compute is C ≈ 6·N·B·S for a batch of B tokens and S steps, reported in petaflop/s-days of 8.64 × 10^¹⁹ operations. Counting embeddings made the trends visibly messier; excluding them puts models of different depth on one curve.
3. Sweep size and shape. Models from 768 to 1.5B non-embedding parameters, varying depth, width, head count and feed-forward size. At fixed N, loss moved only a few percent across a wide range of shapes, so shape could be dropped and N used alone.
4. Train every run with one recipe. Adam, or Adafactor for the models above 1B parameters to save memory, for 2.5 × 10^⁵ steps at 512 sequences of 1,024 tokens, with a 3,000-step linear warmup and cosine decay to zero. The authors report final loss to be largely insensitive to the schedule, as long as there is a warmup, a final decay and a large enough summed learning rate. The schedule length was the same for every run, a choice later work argues about.
5. Measure three single-resource laws. L(N): train each size to near convergence on the full data. L(D): train on subsets from 22M to 23B tokens and stop when test loss stops falling. L(C): take the lower envelope of the loss-versus-compute curves of all sizes.
6. Measure the critical batch size. Using the gradient noise scale of McCandlish and colleagues (2018), find the batch size above which larger batches stop buying a proportional cut in steps. It depends on the loss, not on the model size, and grows as loss falls, reaching about 1 to 2 million tokens near convergence for the largest models trained.
7. Correct for batch size. Convert each run to S_min and C_min, the steps and compute it would need at a batch far below the critical size, so runs at different batch sizes can be compared.
8. Fit two-variable laws and solve for the optimum. Fit L(N, D) to early-stopped runs and L(N, S_min) to learning curves, then minimize loss at fixed C_min. The answer: N ∝ C^^0.73, batch ∝ C^^0.24 and steps ∝ C^^0.03, so almost all extra compute goes into model size.

The two findings the allocation rests on

Larger models are more sample-efficient: they reach a given loss in fewer steps and on fewer tokens than small ones. And the overfitting penalty depends on N^^0.74/D, so each 8× increase in model size needs only about 5× more data to avoid it. Together, the cheapest path down the curve is a much bigger model, trained on modestly more data and stopped well short of convergence.

• N, D — non-embedding parameters and training tokens; Nc, Dc — fitted scale constants, about 8.8 × 10^¹³ parameters and 5.4 × 10^¹³ tokens in the single-variable fits.
• αN ≈ 0.076, αD ≈ 0.095 — the exponents. The ratio α_N/α_D inside the bracket, about 0.74 in the joint fit, is where the N^^0.74/D overfitting rule comes from.
• Smin — the steps a run would need at a very small batch; Sc ≈ 2.1 × 10^3, αS ≈ 0.76.
• Bcrit — the critical batch size, with B* ≈ 2 × 10^8 tokens and αB ≈ 0.21. Training at B_crit costs about twice the minimum steps and twice the minimum data.

At the extremes: as D → ∞ the first law reduces to L(N), and as N → ∞ it reduces to L(D). As S_min → ∞ the second reduces to the converged L(N). Because 1/α_B ≈ 4.8, the critical batch roughly doubles each time the loss falls by about 13%. The constants depend on the tokenizer and data; the authors do not give them a fundamental meaning.

Every number below comes from one model family on one dataset, WebText2, with test loss in nats per token.

Three findings beyond the fits. Loss on other text distributions, such as books, Wikipedia and Common Crawl, tracked WebText2 test loss with a roughly constant offset, so the laws transfer across data in shape if not in level. At a fixed non-embedding parameter count, depth, width and head count changed loss by only a few percent. And LSTMs matched Transformers on tokens early in the context but fell behind on later ones, so the advantage of the architecture is in using long context.

Where the laws break, by their own account

Extrapolated far enough, the compute frontier predicts a lower loss than L(D) allows for the data a compute-efficient run would see, which is impossible. The two trends cross at roughly 10^⁴ PF-days, 10^¹² parameters and 10^¹² tokens, at a loss near 1.7 nats per token; the authors read this as a sign the laws must bend before then. The allocation itself was later revised: Hoffmann and colleagues (2022) found that parameters and data should grow at about the same rate. Why the two studies differ is still argued. Hoffmann and colleagues pointed to the single schedule length; Pearce and Song (2024) to counting only non-embedding parameters at small scale; Porian and colleagues (2024) to uncounted last-layer compute, warmup length and optimizer settings tuned at one scale, and they found careful learning-rate decay was not needed to recover the later result.

The paper is explicit that its power laws are empirical fits with no theoretical derivation, that they must break down eventually — loss cannot fall below the entropy of the data — and that it cannot say where. It also states that its coefficients are specific to the architecture, tokenizer and optimiser settings used, and warns against extrapolating them across such changes.

How it held up, and this is the whole reason the ladder is ordered this way: the method held up and the recommendation did not. Power-law scaling of loss with compute is one of the most robust empirical findings in the field and is still how frontier runs are planned. The parameters-over-data allocation was overturned within two years by rung 5, which found the two should grow at roughly equal rates.

The explanation for the discrepancy is itself instructive and is widely accepted but still contested in detail by later work: the learning-rate schedule in these experiments was not matched to the length of each individual run, which systematically disadvantaged the longer-training, smaller-model configurations — exactly the configurations rung 5 finds are optimal. A detail of the training setup, not of the theory, produced a field-wide misallocation of compute for two years. That is the most useful thing on this page.

Scaling laws are the intellectual license for the entire modern-LLM era. They gave labs the confidence — and the numbers — to commit to nine-figure training runs, because the payoff could be estimated in advance. The chain runs straight from this paper to GPT-3 and every frontier model after it.

They also reframed research priorities. If loss falls predictably with scale, then compute and data are first-class levers, sometimes beating architectural cleverness. A lot of the last few years' progress is, bluntly, this paper's curves being ridden upward.

Scaling laws measure loss, not usefulness. Lower next-word error correlates with capability, but the mapping is loose — a big drop in loss can yield a small or a surprising jump in real skill (emergent abilities). The laws forecast the curve, not exactly which abilities appear where on it.

They also have boundaries: the exponents depend on architecture and data quality, the specific compute-optimal ratio was revised by Chinchilla, and no power law can extend forever — eventually data runs out or the curve must bend. And "just scale" says nothing about alignment, safety, or truthfulness, which don't come for free with size.

This is the paper that turned scaling from a vibe into a discipline. Every time a lab announces a bigger model with a straight face about expected gains, they're invoking scaling laws. The practice of fitting small-scale curves and extrapolating to plan a flagship run is now standard operating procedure across the industry.

More deeply, it changed what the field believes. The "scaling hypothesis" — that a large fraction of intelligence-like behavior falls out of scale applied to a simple objective — got its empirical backbone here. Whether that belief is exactly right is still argued, but it has driven the trajectory of AI more than almost any single idea since the Transformer itself.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →