The GPT-4 report,
explained.

Training a frontier model is one of the largest capital commitments in technology — months of time on tens of thousands of accelerators. Historically you found out whether it worked after you paid. The GPT-4 report's most important methodological claim flips that: OpenAI reported predicting GPT-4's final loss from a series of models trained with up to ~1000× less compute, and landing close to the mark.

This is predictable scaling. It rests on the scaling laws: a model's loss falls off as a smooth power law in training compute, and that curve is stable over many orders of magnitude. If it's that smooth, you don't need the giant run to know where it lands — you fit the curve on cheap small runs and extrapolate. The rest of the report is redacted; this idea is the part that changed how frontier labs work.

The reason extrapolation works is the shape of the loss curve. Across compute scales, loss follows a power law with an irreducible floor: a piece you can never remove (the inherent unpredictability of language) plus a reducible piece that shrinks as a power of compute. Plot loss against compute on log-log axes and the reducible part is a straight line — and a straight line is the easiest thing in the world to extend.

That straightness is what makes a 1000× extrapolation credible. A power law has no wiggles to surprise you: fit its slope and intercept on a handful of points and the line keeps going. The catch is that the fit must be on the reducible loss, so you need the floor — either known or fit jointly — before the log-log line appears. Get that right and the curve becomes a ruler you can lay across three orders of magnitude of compute.

Predictable scaling turns model development from a gamble into a plan. Before committing to the full run, you validate the recipe and infrastructure at small scale, fit the scaling curve, and confirm the final loss will land where you need it. If the small runs don't sit on a clean line, you've found a bug cheaply instead of expensively.

It also lets you compare design choices at small scale and trust that the ranking holds at large scale — a much cheaper way to iterate than running everything at full size. This is how frontier labs now operate: a scaling study of small models de-risks the one enormous run that matters. The expensive run becomes the confirmation of a prediction, not an exploration into the unknown.

Write the loss as an irreducible floor plus a power-law reducible term in compute C:

L_∞ is the irreducible floor; a·C^−b is the part compute buys down. As C grows, the second term vanishes and L approaches L_∞.

Subtract the floor and take logs — the reducible loss is linear in log-compute, so a straight-line fit recovers the exponents:

Because the relationship is a straight line in log-log space, the fit from small-compute runs extends across a huge gap with tiny error — that's why a prediction made from ~1000× less compute can be trusted. The runnable version below fits the law on small runs and nails the loss of a run using 1000× more compute.

The report is thin on model details but rich in results and method:

The report also predicted performance on a specific coding benchmark (HumanEval) by extrapolating from smaller models — a capability, not just a loss — showing prediction can sometimes reach downstream skills. But it candidly flagged the opposite too, which is where the honest limits of the method appear.

Predictable scaling is powerful precisely because it's about loss — a smooth, aggregate quantity. Individual capabilities are dicier. Some appear abruptly at scale (emergent abilities), and the GPT-4 report itself flagged a task where performance ran opposite to the small-scale trend — a case of inverse scaling. So while the report predicted its overall loss beautifully, it was candid that predicting which specific skills a given loss unlocks remains unsolved.

That tension defines the frontier. Loss is a ruler; capabilities are a surprise. The lasting influence of the GPT-4 report isn't a specific number — it's the operating model it legitimized: treat a frontier run as a forecast to be validated, not a mystery to be discovered, and invest in scaling studies so the one giant run is a near-sure thing. Every serious lab now builds this way. The report's redactions made headlines; its scaling methodology quietly reshaped how big models get built.