Let's verify
step by step.

Every rung below rewards an outcome. Rung 8's filter asks only whether the final answer is right, and it admits that this accepts correct answers reached by bad reasoning. This rung's change is to supervise the process: label each step of a solution as sound or not, and train a reward model that scores steps rather than conclusions.

The immediate use is selection. Sample many candidate solutions and pick one — an outcome reward model can only judge the final line, while a process reward model can reject a solution whose fourth step is nonsense however plausible its conclusion looks. That difference compounds with the number of samples, and this is the paper's sharpest empirical result: as you sample more candidates, the gap in favour of process supervision widens. The reason is worth internalising, because it generalises past this paper — the more you sample, the more likely that the best-looking-by-outcome candidate is a fluke, so a coarse judge gains less from a bigger choice set than a step-level one does, and the gap between them widens.

Stated in units: a step-level human-labelled dataset on the order of 800,000 labels, against rung 8's answer key, which costs nothing. That is the trade this rung proposes — replace a free filter that measures the wrong thing with a very expensive one that measures the right thing. The secondary argument is about alignment rather than accuracy: rewarding a correct chain encodes something closer to what you actually want than rewarding a correct conclusion, because only one of those two is what you would ask a person for.

To make a model reason better, you train a reward model to judge its solutions, then use that judge — to rerank many attempts, or to provide a training signal. The question is what the judge looks at. An outcome reward model (ORM) sees only the final answer: right or wrong. A process reward model (PRM) sees every intermediate step and judges each one.

These sound similar but differ profoundly. On a long chain-of-thought solution, the outcome is a single bit at the very end; the process is a rich, step-by-step verdict. And crucially, the final answer is a noisy label for reasoning quality — a chain can be riddled with errors that happen to cancel, landing on the correct number. Grade only the outcome and you teach the model that lucky-but-wrong reasoning is fine. This paper set out to measure how much that matters.

A PRM assigns each step of a solution a correctness score. To turn per-step scores into a solution-level judgment, the natural rule is strict: a solution is correct only if every step is correct, and its confidence is roughly the product of the per-step correctness probabilities. Contrast the two judges:

Because the product of many sub-1 probabilities shrinks, a single doubtful step drags the whole solution's score down — exactly the behavior you want. Two things fall out for free: the PRM localizes the first mistake (the step where correctness breaks), and it rejects "right answer, wrong reasoning" chains that an ORM would happily reward. The runnable version below shows a lucky flawed chain passing the ORM but failing the PRM.

Beyond catching lucky answers, process supervision helps for reasons that compound:

There's also an alignment argument the paper highlights: process supervision directly rewards reasoning a human would endorse at each step, rather than an outcome that might be reached by unfaithful means. Rewarding the process encourages the model to be right for the right reasons — a safer target than rewarding the answer alone, especially as tasks get harder to check.

To train a PRM you need step-level labels, and the paper collected them at scale: PRM800K, roughly 800,000 human annotations marking each step of model-generated MATH solutions as correct, incorrect, or neutral. That dataset — released with the paper — is what makes step-level supervision possible, and its release fueled a wave of follow-up work on reasoning verifiers.

Collecting per-step human labels is expensive, which is the honest cost of process supervision: it needs finer-grained annotation than "is the final answer right." The paper's contribution was partly to show that this cost pays off — the resulting verifier is good enough that, used to rerank many sampled solutions, it lifts accuracy well past outcome-based selection. Later work has explored generating step labels automatically to reduce the human burden.

The paper's question is narrow and testable: if you train one reward model on final-answer correctness and another on step-by-step human labels, which one better picks correct solutions to hard math problems? The large-scale models are fine-tuned from the GPT-4 base model, pretrained only for next-token prediction with no RLHF, after a further pass over MathMix, about 1.5B tokens of math-related text. A small-scale family, pretrained with roughly 200 times less compute, supports controlled experiments.

1. Make a step-by-step generator. The base model is few-shot prompted to solve MATH training problems, the solutions that reach the correct final answer are kept, and the base model is fine-tuned on them for one epoch. The goal is format, not skill: one reasoning step per line, so steps can be labeled and scored separately.
2. Collect step labels (PRM800K). Human labelers mark each step of generated solutions as positive, negative or neutral. After filtering, the dataset holds about 800K step labels across 75K solutions to 12K problems.
3. Label where the model is wrong. A small first phase, about 5% of the data, labeled alternative completions for each step. The second phase ran in 10 generations: the process reward model (PRM) was retrained between generations, and labelers were shown convincing wrong-answer solutions, the ones the current PRM scored highly but whose final answer was wrong. In small-scale experiments, the authors estimate that this kind of active learning is about 2.6 times more data-efficient than labeling uniform samples.
4. Train the PRM. The PRM is a language model trained to predict the label at the end of each step, so one forward pass over a solution yields a correctness probability for every step.
5. Score a whole solution. A solution's score is the probability that every step is correct, computed as the product of its step probabilities. Neutral steps count as correct in the main results.
6. Train the outcome baseline. The outcome reward model (ORM) is trained on 100 uniform samples per problem from the generator, labeled only by whether the final answer is correct. That is about ten times more solutions than PRM800K, with no overlap, and its prediction at the final token is the solution's score.
7. Select best-of-N. For each test problem, sample up to N = 1,860 solutions, keep the one each judge scores highest, and check its answer. Majority voting over final answers is the third method.
8. Run a controlled comparison. Because PRM800K and the ORM data differ in more than their labels, the small-scale experiments use the large PRM as a labeler for small reward models, comparing process supervision, outcome supervision from that same labeler, and outcome supervision by final-answer checking.

The generator was never trained with reinforcement learning against the PRM. Every result is about selection: how well each judge picks from a fixed pool of samples.

For a solution s with K steps:

• pφ(stepk correct | …) — the PRM's probability, read at the end of step k, that the step is correct given everything before it; neutral labels count as correct.
• Π — the product makes the score a weakest-link measure: one doubtful step caps the whole solution.
• pψ — the ORM's probability, read at the final token, that the answer is right; it was trained only on final-answer checks.

At the extremes: with N = 1 there is nothing to choose, so all judges tie. As N grows, a judge's errors matter more, because the selected solution is whichever one it overrates most. An ORM trained on final answers learns that a correct answer reached through faulty steps is good, so it is easiest to fool exactly when the pool contains many plausible-looking solutions. Length matters too: twenty steps at 0.99 each multiply to about 0.82.

The main test set is 500 MATH test problems. The other 4,500 test problems were folded into PRM800K's training data, so the authors held out 500 chosen uniformly at random, which keeps them representative of the full test set. That subset is the benchmark now widely called MATH-500.

The PRM's lead over the ORM widened as N grew. The ORM only slightly beat majority voting overall, and on the easiest fifth of problems its accuracy fell slightly as more samples were added: with more candidates, there are more convincing wrong ones for a coarse judge to pick.

The controlled small-scale experiments, in which the large PRM labeled data for small reward models, removed the worry that PRM800K was simply better data. Process supervision beat both kinds of outcome supervision at every data size tested, and outcome supervision from the large PRM beat final-answer checking, because final-answer labels mark lucky solutions as correct. Active learning made labels about 2.6 times more data-efficient.

Out of distribution, on recent AP Calculus, AP Chemistry, AP Physics and AMC10/12 problems released after the pretraining data was collected, best-of-100 selection solved 72.9% with the PRM, 63.8% with the ORM and 61.3% with majority voting. The authors describe the result as a negative alignment tax: the more interpretable, human-endorsed form of supervision was also the more accurate one. The limits are plain too: every result is mathematics, and nothing here tests training a generator against the PRM.

The cost is the admission, and the size of the dataset is the evidence for it: human step-level annotation of mathematical reasoning is skilled, slow work, and it does not obviously transfer to domains where "a step" is not a well-defined object. The results are on mathematics, where solutions decompose cleanly. A process reward model is also a learned model and therefore gameable in the way rung 1 warned about, and its reliability degrades on reasoning unlike what it was trained on.

What happened next cuts in two directions and both belong here. The expense pushed the field toward automating the step labels — estimating a step's value by sampling completions from it and seeing how often they succeed, which replaces annotators with compute. But the sharper challenge came from the reasoning ladder: a model trained with outcome-only reward at scale developed long, coherent reasoning anyway, without anyone labelling a single step. That is genuinely in tension with this rung's framing, and the tension is unresolved — one reading is that process supervision is a shortcut that enough outcome reward makes unnecessary, another is that it remains the better signal and outcome RL at scale is paying compute to rediscover it.

How it held up: the concept — a verifier that scores intermediate work — is central to how test-time compute is spent, and the finding that a coarse judge gains less than a step-level one as the sample count rises is one of the most useful transferable results on this ladder. Whether the labels should come from people, from a model, or from the eventual reward is still argued about.

Process reward models became a central tool of the reasoning era. The idea — verify reasoning step by step, and use that verifier to select or train — runs directly into the test-time-compute and reasoning-model wave: sampling many chains and reranking with a process verifier is a core recipe, and step-level rewards feed reinforcement-learning pipelines that push models to reason more carefully. Much subsequent work builds PRMs, often trying to generate the step labels automatically to escape the human-annotation cost.

The deeper lesson is about what you measure. It is tempting to grade the thing you ultimately care about — the answer. But when the answer is a lossy, gameable proxy for the quality you actually want (sound reasoning), a denser signal on the process itself is both more informative and better aligned. "Grade the reasoning, not just the result" turned out to be one of the more consequential ideas for making models reason.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →