Let's verify
step by step.

To make a model reason better, you train a reward model to judge its solutions, then use that judge — to rerank many attempts, or to provide a training signal. The question is what the judge looks at. An outcome reward model (ORM) sees only the final answer: right or wrong. A process reward model (PRM) sees every intermediate step and judges each one.

These sound similar but differ profoundly. On a long chain-of-thought solution, the outcome is a single bit at the very end; the process is a rich, step-by-step verdict. And crucially, the final answer is a noisy label for reasoning quality — a chain can be riddled with errors that happen to cancel, landing on the correct number. Grade only the outcome and you teach the model that lucky-but-wrong reasoning is fine. This paper set out to measure how much that matters.

A PRM assigns each step of a solution a correctness score. To turn per-step scores into a solution-level judgment, the natural rule is strict: a solution is correct only if every step is correct, and its confidence is roughly the product of the per-step correctness probabilities. Contrast the two judges:

Because the product of many sub-1 probabilities shrinks, a single doubtful step drags the whole solution's score down — exactly the behavior you want. Two things fall out for free: the PRM localizes the first mistake (the step where correctness breaks), and it rejects "right answer, wrong reasoning" chains that an ORM would happily reward. The runnable version below shows a lucky flawed chain passing the ORM but failing the PRM.

Beyond catching lucky answers, process supervision helps for reasons that compound:

There's also an alignment argument the paper highlights: process supervision directly rewards reasoning a human would endorse at each step, rather than an outcome that might be reached by unfaithful means. Rewarding the process encourages the model to be right for the right reasons — a safer target than rewarding the answer alone, especially as tasks get harder to check.

To train a PRM you need step-level labels, and the paper collected them at scale: PRM800K, roughly 800,000 human annotations marking each step of model-generated MATH solutions as correct, incorrect, or neutral. That dataset — released with the paper — is what makes step-level supervision possible, and its release fueled a wave of follow-up work on reasoning verifiers.

Collecting per-step human labels is expensive, which is the honest cost of process supervision: it needs finer-grained annotation than "is the final answer right." The paper's contribution was partly to show that this cost pays off — the resulting verifier is good enough that, used to rerank many sampled solutions, it lifts accuracy well past outcome-based selection. Later work has explored generating step labels automatically to reduce the human burden.

The headline is a clean win for process supervision on hard reasoning:

The reranking result is the practical punchline: sample many chains, score each with the PRM, keep the best. Because the PRM judges reasoning rather than luck, that selection lands on correct, well-reasoned solutions much more reliably than picking by a final-answer verifier.

Process reward models became a central tool of the reasoning era. The idea — verify reasoning step by step, and use that verifier to select or train — runs directly into the test-time-compute and reasoning-model wave: sampling many chains and reranking with a process verifier is a core recipe, and step-level rewards feed reinforcement-learning pipelines that push models to reason more carefully. Much subsequent work builds PRMs, often trying to generate the step labels automatically to escape the human-annotation cost.

The deeper lesson is about what you measure. It is tempting to grade the thing you ultimately care about — the answer. But when the answer is a lossy, gameable proxy for the quality you actually want (sound reasoning), a denser signal on the process itself is both more informative and better aligned. "Grade the reasoning, not just the result" turned out to be one of the more consequential ideas for making models reason.