The GPT-4 report,
explained.

Rung 1 reports what a very large model turned out to be able to do, which is how the field worked: commit an enormous budget, train, and find out. This rung's contribution is methodological and is the reason it is on the ladder at all — you can predict the outcome before you spend the money.

The method is to train a series of much smaller models, fit a power law to how their loss falls with compute, and extrapolate to the run you intend to do. The report describes predicting the final loss of the large run from models trained with up to around a thousand times less compute, and landing close. It also does the harder version: predicting not just loss but performance on a capability benchmark, which requires the extra assumption that the loss-to-capability relationship is itself regular.

Why that reframes the economics. A frontier training run is a single irreversible expenditure at a scale where a failure is material to the organisation. If the outcome is forecastable from cheap experiments, the run stops being a bet and becomes a plan — you can decide whether the resulting capability is worth the money before committing, and you can compare recipes at small scale with some confidence the ordering survives. This is the practice that made scaling an engineering discipline rather than a series of expensive surprises.

Training a frontier model is one of the largest capital commitments in technology — months of time on tens of thousands of accelerators. Historically you found out whether it worked after you paid. The GPT-4 report's most important methodological claim flips that: OpenAI reported predicting GPT-4's final loss from a series of models trained with up to ~10,000× less compute, and landing close to the mark (the HumanEval pass-rate forecast used models with up to ~1,000× less).

This is predictable scaling. It rests on the scaling laws: a model's loss falls off as a smooth power law in training compute, and that curve is stable over many orders of magnitude. If it's that smooth, you don't need the giant run to know where it lands — you fit the curve on cheap small runs and extrapolate. The rest of the report is redacted; this idea is the part that changed how frontier labs work.

The reason extrapolation works is the shape of the loss curve. Across compute scales, loss follows a power law with an irreducible floor: a piece you can never remove (the inherent unpredictability of language) plus a reducible piece that shrinks as a power of compute. Plot loss against compute on log-log axes and the reducible part is a straight line — and a straight line is the easiest thing in the world to extend.

That straightness is what makes a 10,000× extrapolation credible. A power law has no wiggles to surprise you: fit its slope and intercept on a handful of points and the line keeps going. The catch is that the fit must be on the reducible loss, so you need the floor — either known or fit jointly — before the log-log line appears. Get that right and the curve becomes a ruler you can lay across three orders of magnitude of compute.

Predictable scaling turns model development from a gamble into a plan. Before committing to the full run, you validate the recipe and infrastructure at small scale, fit the scaling curve, and confirm the final loss will land where you need it. If the small runs don't sit on a clean line, you've found a bug cheaply instead of expensively.

It also lets you compare design choices at small scale and trust that the ranking holds at large scale — a much cheaper way to iterate than running everything at full size. This is how frontier labs now operate: a scaling study of small models de-risks the one enormous run that matters. The expensive run becomes the confirmation of a prediction, not an exploration into the unknown.

Write the loss as an irreducible floor plus a power-law reducible term in compute C:

L_∞ is the irreducible floor; a·C^^−b is the part compute buys down. As C grows, the second term vanishes and L approaches L_∞.

Subtract the floor and take logs — the reducible loss is linear in log-compute, so a straight-line fit recovers the exponents:

Because the relationship is a straight line in log-log space, the fit from small-compute runs extends across a huge gap with tiny error — that's why a prediction made from ~10,000× less compute can be trusted. The runnable version below fits the law on small runs and nails the loss of a run using 1000× more compute.

The report withholds architecture, size, hardware, compute, data and training method, so what can be described is the forecasting procedure, the alignment pipeline and the evaluation protocol.

1. Make the stack predictable first. The authors say a large focus of the project was infrastructure and optimization methods that behave predictably across scales, because a run the size of GPT-4 cannot be tuned model by model. Predictability is the precondition for the forecast, not a by-product of it.
2. Train small models the same way. A family of runs used the same methodology at a fraction of the compute: up to 10,000 times less for the loss forecast, and up to 1,000 times less for the capability forecast.
3. Forecast the final loss. Fit L(C) = aC^^b + c, a power law with an irreducible term, to the small runs' final loss on an internal code dataset that is not in the training set. The prediction was made shortly after the GPT-4 run started, without using any of its partial results, and it matched the final loss with high accuracy.
4. Forecast a capability. For HumanEval, fit the mean log pass rate as a power law in compute. Very low pass rates cannot be estimated, so the fit uses only problems that every small model solves at least once given a large sampling budget. The 15 hardest problems were left out, the rest were split into six difficulty buckets by small-model performance, and the predictions were registered before training finished. On the third-easiest bucket they were very accurate; on the easiest, GPT-4 came in below the prediction.
5. Pretrain, then fine-tune. GPT-4 is a Transformer-style model pretrained to predict the next token on public and licensed data, most of it from before September 2021, and it accepts images as well as text. It was then fine-tuned with reinforcement learning from human feedback.
6. Add safety rewards. Safety-relevant prompts are added to RLHF, and rule-based reward models supply an extra reward. Each RBRM is a zero-shot GPT-4 classifier given the prompt (optionally), the policy's output and a human-written rubric, which sorts the output into categories such as a refusal in the desired style, a refusal in an undesired style (evasive or rambling), disallowed content, or a safe non-refusal. The policy is rewarded for refusing harmful requests and for not refusing harmless ones.
7. Evaluate with contamination checks. After stripping spaces and symbols, three random 50-character substrings of each test item are searched for in the training data; matching items are removed and the scores rerun. Portions of BIG-bench turned out to be in the training set, so it was left out of the results, and part of the GSM-8K training set was deliberately mixed into pretraining.
8. Red-team. More than 50 experts in fields such as long-term AI alignment risks, cybersecurity, biorisk and international security tested the model adversarially before release; the detail is in the accompanying system card.

• C — training compute, normalized so GPT-4's run is 1; the small runs sit at fractions of it.
• a, b — the fitted scale and slope of the reducible loss; c — the irreducible loss the curve approaches but never crosses.
• P — a subset of HumanEval problems; pass_rate(C) — the fraction of sampled programs that pass a problem's tests, for a model trained with compute C.
• α, k — the fitted constants of the capability law.

Taking the log before averaging is what makes the second law work: a problem going from a 1-in-1,000 to a 1-in-100 pass rate counts as much as one going from 10 to 100 percent, so progress on hard problems is visible long before they are routinely solved. It is also why problems that some model never solves had to be excluded, since the log of zero is undefined. At the limits, as C grows the loss approaches c and the left side of the second law approaches 0, a pass rate of 1. Neither law can bend back upward, so a task whose performance falls with scale and then recovers, like Hindsight Neglect, lies outside what either can predict.

Exams were taken by the RLHF model under simulated conditions, with contaminated questions removed; academic benchmarks compare GPT-4 with GPT-3.5 and the best published results.

On MMLU translated into 26 languages, GPT-4 beat the English-language state of the art in 24 of them, and beat the English scores of GPT-3.5, Chinchilla and PaLM in most languages tested, including low-resource ones such as Latvian, Welsh and Swahili. Exam results appear to come mainly from pretraining: on multiple-choice questions the base and RLHF models scored about the same on average.

Post-training had measurable costs and benefits. On a subset of MMLU the pretrained model was highly calibrated, with an expected calibration error of 0.007; after PPO it was 0.074. The safety pipeline cut responses to requests for disallowed content by 82 percent relative to GPT-3.5, and responses to sensitive requests such as medical advice and self-harm followed policy 29 percent more often. On internal adversarial factuality evaluations GPT-4 scored 19 points above the latest GPT-3.5.

The forecasting section includes its own counterexample. On Hindsight Neglect, an Inverse Scaling Prize task on which OpenAI's smaller models got worse as they grew, GPT-4 reversed the trend. The report draws the conclusion itself: certain capabilities remain hard to predict, and the method forecasts loss and a well-behaved benchmark subset, not which skills a given loss will bring.

The report is explicit that it withholds almost everything: no architecture, no model size, no data, no training details, citing competition and safety. That is a deliberate break with the field's publishing norms and it was widely criticised as such, including for the word "technical report" doing work the contents do not support. It is a capability and evaluation document; the one methodological claim it does make is the one above.

Where prediction breaks down is also acknowledged rather than glossed. Loss is much more predictable than capability, and individual task performance can move non-monotonically — including tasks that get worse with scale, which the extrapolation has no way to anticipate. So the claim is bounded: the aggregate is forecastable, the specifics are not, and anything that emerges sharply is by construction outside what a smooth fit can tell you.

How it held up: predictable scaling became standard internal practice and is now assumed rather than argued for. The other thing that spread was the non-disclosure: subsequent frontier releases largely followed this template, and the gap this site has to work with — architectural detail available for open models and absent for the strongest closed ones — dates from here. Both halves of the legacy are real, and only one of them was a contribution.

Predictable scaling is powerful precisely because it's about loss — a smooth, aggregate quantity. Individual capabilities are dicier. Some appear abruptly at scale (emergent abilities), and the GPT-4 report itself flagged a task where performance ran opposite to the small-scale trend — a case of inverse scaling. So while the report predicted its overall loss beautifully, it was candid that predicting which specific skills a given loss unlocks remains unsolved.

That tension defines the frontier. Loss is a ruler; capabilities are a surprise. The lasting influence of the GPT-4 report isn't a specific number — it's the operating model it legitimized: treat a frontier run as a forecast to be validated, not a mystery to be discovered, and invest in scaling studies so the one giant run is a near-sure thing. Every serious lab now builds this way. The report's redactions made headlines; its scaling methodology quietly reshaped how big models get built.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Four papers that are reports, not methods: they state capability and withhold recipe. Read for the evaluation design and the omissions, not the architecture.

1. 1PaLM2022
2. 2GPT-4 Technical Report2023
3. 3Gemini 1.52024
4. 4GPT-5.6 System Card2026

All 110 papers, in 19 ladders →