Self-improving
agents.

You ask the model once, you get one answer, and you judge the model on it. That is a strange basis for a judgement, because a model is a sampler. Ask the same question again and something else comes out. Ask it 250 times and the question quietly changes from can this model do the task to does this model ever do the task.

Those two questions have very different answers. In Large Language Monkeys: Scaling Inference Compute with Repeated Sampling (Brown et al., 2024), the authors draw up to 10,000 independent samples per problem across five tasks and track coverage: a problem counts as covered the moment one of its samples is correct, no matter which one and no matter what the other 9,999 said. Coverage climbs, smoothly, over four orders of magnitude of sample budget.

The result that should reorganise how you think about agents is on SWE-bench Lite, where a problem is a real GitHub issue and one "sample" is an entire multi-turn trajectory through a codebase. DeepSeek-Coder-V2-Instruct, driving the open-source Moatless Tools agent framework, resolves 15.9% of issues on a single attempt. Give it 250 independent attempts and 56% of issues are resolved by at least one of them. The single-attempt state of the art at the time — CodeStory Aide, running a mixture of GPT-4o and Claude 3.5 Sonnet — was 43%. Carry one condition with those numbers: every attempt, the single one included, was drawn at temperature 1.6, chosen by a sweep over 1.0, 1.4, 1.6 and 1.8 on 50 problems. That is a setting picked for the many-attempts regime, not DeepSeek's best single-shot configuration.

Brown et al. 2024, SWE-bench Lite via Moatless Tools · attempts drawn independently at temperature 1.6, no feedback between them

Read that slowly, because it is the premise the rest of this page is built on: a weaker model, sampled more, found fixes a stronger model did not find on its first try. The capability was already in the weights. What was missing was attempts, and a temperature loose enough to make them differ.

It is also cheaper than it sounds. At the API prices current when the paper was written, a much smaller version of the same trade — five DeepSeek attempts, with an oracle picking the winning patch, against one attempt each from GPT-4o and Claude 3.5 Sonnet — looks like this:

Brown et al. 2024, Table 1 · five cheap attempts solve more issues than one expensive attempt, at 3.6x less than GPT-4o and 4.7x less than Claude 3.5 Sonnet

And it is not a frontier-model effect. Gemma-2B on CodeContests goes from a 0.02% solve rate at one sample to 7.1% at 10,000 — more than a 300x increase. Pythia-160M goes from 0.27% to 57% on a 128-problem MATH subset — coverage under an oracle answer checker rather than a solve rate, for the reason the next paragraph gives. Small models contain far more correct answers than they can produce on demand.

Say that without any of the numbers attached, because it is the premise the whole page rests on: a model’s first answer is not the edge of what it knows. Everything that follows is about the distance between those two things, and who closes it.

Now the part that decides whether you build this well or badly. Coverage is not accuracy. Coverage counts a problem as solved if any of your samples was correct, which quietly assumes something you usually do not have: an oracle that can look at 10,000 candidate answers and point to a right one.

Where that oracle is real, coverage converts directly into performance. A Lean proof checker either accepts the proof or it does not. A test suite either goes green or it does not. That is exactly why SWE-bench Lite works as a demonstration — the harness grades every attempt with real tests. But those tests are withheld from the agent, so the 56% is what an oracle choosing among 250 patches scores, not what a shippable selector scores, and Brown et al. found 11.3% of Lite problems have flaky test suites on top of that. A checker that can actually decide is also why verifiable-reward domains keep producing the loudest results in this literature.

Where the oracle is not real, the gap is enormous, and Brown et al. measured it. On 128 MATH problems with Llama-3-8B-Instruct at 10,000 samples each, coverage rises from 79.8% at 100 samples to 95.3% at 10,000. Over that same range, the methods you could actually ship — majority voting and reward-model selection, using ArmoRM-Llama3-8B-v0.1, then the highest-scoring open-weight model for reasoning on RewardBench — move from 38.7% to 39.8%. Every selection method saturates before about 100 samples while coverage keeps climbing. At 10,000 samples the best available selector is leaving more than 55 points of already-generated, already-correct answers on the floor.

Stop on that for a moment. The samples got better and the answer you would have shipped did not. Buying more attempts only helps if something can tell you which attempt to keep — and that something is now the expensive part of the system, not the model.

Brown et al. 2024 · Llama-3-8B-Instruct, the paper's 128-problem MATH subset, up to 10,000 samples per problem · the plotted endpoints are the numbers the paper reports; anything drawn between them is interpolation, not a measurement

The reason is mechanical rather than mysterious. On some of those problems the correct answer is produced by 1% of samples or fewer. A majority vote over 10,000 samples cannot surface a 1% answer — the more you sample, the more precisely the vote converges on the popular wrong one. Sampling finds needles; voting measures the haystack. Closing that gap is what learned verifiers are for, and why their failure modes matter so much here.

This gap is not new, and the first clean receipt for it is five years old. In the Codex paper (Chen et al., 2021), Codex-S solves 37.7% of HumanEval problems with one sample at temperature 0.8. Given 100 samples per problem, picking the candidate with the highest mean log-probability solves 44.5%; picking the candidate that passes the unit tests solves 77.5%. Same 100 samples, same model — 33 points between the best ranker and the oracle. AlphaCode is the same shape at production scale, with the selector built out in public: it draws an enormous pool of candidate programs, throws away every one that fails the example tests printed in the problem statement (roughly 99% of the pool), runs what survives on generated inputs, and groups the programs that produce identical outputs — so its ten allowed submissions go to ten different behaviours rather than ten spellings of one. It reached an average rank in the top 54.3% across simulated Codeforces competitions with more than 5,000 participants.

You cannot steer a loop on a number you are measuring wrong, and coverage is easy to measure wrong in both directions.

Start with what you actually want to know. pass@k is the probability that if you drew k samples for a problem, at least one would be correct. It is a property of the model-and-problem pair, not of one particular run of your harness.

The obvious way to estimate it: draw k samples, check whether any passed, write down 1 or 0, average over problems. That is unbiased and nearly useless. For a single problem it is one Bernoulli draw, so its variance is about as large as the thing you are measuring — the high variance that made Chen et al. reach for something better. It is also wasteful — pass@1, pass@10 and pass@100 would each need their own experiment.

The version that actually ships is worse. Generate a big pool of n samples, notice that one of them worked, and report that as pass@k for whatever k you had in mind. That is pass@n wearing a smaller label. With 200 samples of which 2 are correct, "did any of them work" reports 1.0 while the true pass@10 is 0.098 — a factor of ten, in the direction that flatters you.

The Codex paper (Chen et al., 2021), which introduced HumanEval, fixed both problems with one formula. Generate n ≥ k samples once, count the c that pass, then ask a combinatorial question of the pool you already have:

C(n−c, k) / C(n, k) is the probability that a k-subset drawn from your n samples without replacement contains none of the c correct ones. One minus that is the probability it contains at least one. The Codex paper used n = 200 and k ≤ 100; Brown et al. reuse this same estimator to keep variance down at 10,000 samples.

The economy is the point. One pool of n samples gives an unbiased estimate at every k ≤ n, so a single expensive run draws the whole curve instead of one point on it.

There is a near-miss worth naming, because it looks right and it is not. Take p̂ = c/n and write pass@k ≈ 1 − (1 − p̂)^k. That treats your k draws as being made with replacement from the pool, and Chen et al. show in their appendix that it is a consistent underestimate whose gap does not fully close even when n > 5k. It is wrong in the modest direction — it makes sampling look less useful than it is — which is exactly why nobody catches it.

Three edge cases carry all the meaning, and the code below is where you write them:

• k > n. Undefined; the function should refuse rather than guess. You cannot estimate pass@1000 from 200 samples no matter how straight the curve looks.
• c = 0. The estimator returns exactly 0 at every k. Read it as "not yet", never as "impossible" — 200 consecutive failures only put a rough 95% upper bound of p < 3/200 on the true rate, and section 03 is about how much that tail matters.
• c = n. Returns exactly 1, as does any case where n − c < k: if fewer than k of your samples are wrong, no k-subset can be all wrong. Chen et al.'s implementation short-circuits that case before the product runs — not because anything would divide by zero, but because the product would otherwise grind through factors like 1 − k/j that go negative.

Then the caveat that outranks the arithmetic: c is whatever your checker counted. Brown et al. ran CodeContests' own test suites against the reference Python3 solutions the dataset ships, and found that of the 122 test-set problems with such solutions, 35 have correct solutions that fail their own tests. Those are false negatives — coverage you measured as absent and in fact had. The mirror failure, a wrong program that passes a thin test suite, inflates the same number instead. What you are reporting is never pass@k; it is passes-my-verifier@k, which is why how you build the evaluator and where you run it are load-bearing decisions rather than plumbing.

Plot coverage against sample budget with a logarithmic x-axis and you usually get something suspiciously tidy: a line. Brown et al. observe coverage growing "nearly log-linearly" across several orders of magnitude for the Llama-3 and Gemma families — with exceptions they flag, such as Llama-3-8B-Instruct on MiniF2F-MATH — and fit it with an exponentiated power law, modelling log(coverage) ≈ a·k^−b, then exponentiating to predict coverage directly.

The tempting reading is that the slope is a fact about the model, the way parameter count is. It is not. It is a fact about the distribution of difficulty in your problem set, and the derivation is four lines long.

Start with one problem. Your agent solves it with probability p on any single attempt, and attempts are independent, so the chance that all k fail is (1 − p)^k:

Exponential in k. On a log-x plot this is a sharp S — nothing, nothing, then everything, almost all of it inside two decades around k ≈ 1/p. A single problem's curve is not straight anywhere.

Now the benchmark. Coverage is that quantity averaged over problems, which means averaging over whatever spread of per-problem success rates your problem set happens to contain:

Every problem is being solved exponentially fast. The benchmark is a weighted sum of those exponentials, and the weights are f — the density of single-attempt success rates across your problems. A weighted sum of exponentials is precisely where power laws come from.

How Do Large Language Monkeys Get Their Power (Laws)? (Schaeffer et al., ICML 2025) makes that exact. They prove it in both directions. If the density behaves like a power law near zero — f(p) = C·p^b−1 for small p — then for large k, −log(coverage) ~ C·Γ(b)·k^−b. And conversely: if −log(coverage) ~ A·k^−b, the density must satisfy f(p) ~ (A/Γ(b))·p^b−1 as p → 0. In English: the slope of your inference-scaling curve is the exponent of the left tail of your difficulty distribution. Nothing in that derivation is about the model.

Their appendix works the dependence out for named distributions, which makes it concrete:

Schaeffer et al. 2025, Appendix E · the real data fit a 3-parameter scaled Kumaraswamy distribution, because most benchmarks top out well below p = 1

The best evidence for the account is the case it explains away. Not every setting produces a power law. In the Best-of-N jailbreaking results of Hughes et al. (2024), Llama-3-8B-IT broke the pattern every other model followed: its −log(attack success rate) fell faster than any power law, which is to say its success rate climbed faster than any power law allows. Schaeffer et al. account for it from the distribution alone — within the sampling budget, every prompt eventually succeeded, so there was no heavy left tail, so there was no aggregate power law. The straight line requires near-impossible problems to exist. Remove them and it bends.

You can watch the whole argument happen in arithmetic. Nothing below runs a model or samples anything: it averages 1 − (1 − p)^k over two hand-built difficulty distributions of 128 problems each — one where every problem is equally hard, one with a power-law left tail — whose mean single-attempt success rate is identical to four decimals. The shape is a property of those two distributions and of nothing else.

Three consequences you can act on:

• Measure the tail, not the mean. Two workloads with identical pass@1 can have opposite returns on sampling, which the code above demonstrates with two 128-problem sets whose average difficulty is the same to four decimals. Before budgeting for repeated sampling, look at the histogram of per-problem success rates, not the average.
• You can forecast the curve without paying for it. Because the exponent lives in f, you can fit f from cheap single-attempt measurements and simulate the rest. Schaeffer et al. report roughly an order of magnitude lower relative error on the exponent than log-log least squares — measured by backtesting both estimators on synthetic data with a known ground-truth exponent — which they put at about 2–4 orders of magnitude less inference compute for the same precision.
• When the line bends, ask which tail went missing. Saturation means you have run out of problems that are hard but possible. Either the stragglers sit at p = 0 — a capability the model genuinely lacks, or a verifier that is rejecting correct work — or you have solved everything you had. Those two want opposite responses, and only the histogram tells you which one you are looking at.

Sampling k times is the bluntest way to spend inference compute, and the curve above is its price list. For the rest of the menu — longer reasoning, search, verification passes, and how to divide a fixed budget between them — see test-time compute. The remainder of this page takes the other road: keep what the sampling found, and move it into the model so the first attempt is the good one.

Everything so far assumed a step you do not have: the ability to say that one was good. In a notebook you supply that yourself. In a loop running on production traffic nobody is reading the traces — the agent grades its own homework, at volume, and whatever it grades highly becomes tomorrow's memory or tomorrow's gradient. The oracle is not deployable. Something worse has to stand in for it.

Generating candidates is the cheap half, and it is startlingly cheap. Large Language Monkeys (Brown et al., 2024) measured this directly with a metric they call coverage: the fraction of problems solved by any sample in the batch. Coverage keeps climbing with the sample budget across four orders of magnitude, often log-linearly. On SWE-bench Lite, DeepSeek-Coder-V2-Instruct resolves 15.9% of issues with one sample and 56% with 250 samples — past the 43% single-sample state of the art at the time. That 56% is the model's reach. Whether any of it reaches production depends entirely on the picker, and the same paper reports what pickers actually do: in domains without an automatic verifier, majority voting and reward-model ranking plateau beyond several hundred samples and fail to keep scaling with the sample budget.

So the binding constraint on a self-improvement loop is not can the agent do it. It is can you recognise that it did. Test-time compute covers how to spend the sampling budget; this section is about the pile you are left holding afterwards. Three selectors work in practice. Each one breaks somewhere specific, and the place it breaks is exactly where your loop learns the wrong thing — quietly, and on repeat.

Majority vote: cheap, strong, and pointed at the wrong target

Sample many reasoning chains, throw away the reasoning, keep the final answers, take the mode. That is self-consistency (Wang et al., 2022). It costs N generations and a counter; the table below is what that buys.

Wang et al. sampled 40 reasoning paths per question and marginalised the paths out. Read the formula again: it converges on the model's most frequent answer. Nothing in it points at the truth.

It works, and the size of the effect is why everyone reaches for it first. On GSM8K, with 40 sampled paths against greedy chain-of-thought:

Across the paper's other benchmarks the abstract's headline gains are SVAMP +11.0, AQuA +12.2, StrategyQA +6.4 and ARC-challenge +3.9 — quoted there without a model attached, so read them as the paper's best case rather than as PaLM-540B's row extended sideways. Read the table down the model column, though, and the shape of the thing shows: the weaker the model, the smaller the gain. A selector amplifies a skew that is already there. It cannot create one.

Two places it breaks.

It needs an extractable final answer. This is not a limitation someone found later — it is stated in the paper. Self-consistency applies only where the final answer comes from a fixed answer set, and the authors say plainly that extending it to open-text generation would first require defining a good metric of consistency between outputs. So: no code, no prose, no multi-step tool trajectory, no pull request. Everything an agent actually produces is out of scope for the cheapest selector you have. That is not a footnote. That is the reason the rest of this section exists.

It bends downward. In Are More LLM Calls All You Need? (Chen, Zaharia and Zou, 2024), the accuracy of Vote and Filter-Vote can first increase and then decrease as the number of model calls grows. With GPT-3.5-turbo-0125 on the MMLU college mathematics and business ethics subsets, accuracy is maximised at one particular number of calls and gets worse in both directions from there; on college chemistry it stays monotone. Their explanation is the one that matters for agent work: a task is a mixture of easy and hard queries, more calls push easy queries toward the right answer and hard queries toward the wrong one, and the sum of a rising curve and a falling curve is a hump.

The mechanism underneath is worth stating on its own, because it is the thing people get wrong. Majority vote converges on the model's mode. When the mode is correct, more votes sharpen it toward 1. When the mode is wrong, more votes sharpen it toward 0. There is no third case. And wrong answers share a mode more often than you would hope, because a model's errors are not random draws — a misread constraint, an off-by-one in the units, a plausible-but-false lemma. Every sample makes the same slip, because every sample came from the same weights looking at the same prompt. Diversity of sampled paths is not diversity of failure modes. When wrong answers share an attractor, the vote is a machine for becoming confident about it.

The block below computes that hump exactly. Its two difficulty profiles — an easy question the model gets right 70% of the time, a hard one where 65% of samples land on the same wrong answer — are numbers chosen by hand, not measurements. Nothing here was run against a model. The shape of the curve is what to read; the scale is invented.

Reward-model ranking: a score you can over-optimise

The fix for "no extractable answer" is to learn the judgment instead. Train a verifier on solutions labelled correct and incorrect, sample N candidates at inference, score them all, keep the top one. Best-of-N.

The original result is still the clearest. In Training Verifiers to Solve Math Word Problems (Cobbe et al., 2021 — the paper that introduced GSM8K's 8.5K problems), 100 completions are sampled and ranked at test time, and the paper's own summary is that “on the full dataset, 6B verification slightly outperforms a finetuned 175B model, thereby offering a boost approximately equivalent to a 30x model size increase.” Ranking is not a tiebreak. It is a substitute for scale.

Now the break, which is in the same paper and usually gets skipped. For the 6B verifier, performance improves up to 400 completions and past that point starts to decrease. The authors' reading, in their words: “the benefits of search are eventually outweighed by the risk of finding adversarial solutions that fool the verifier.” With enough candidates you are no longer searching for a correct solution. You are searching for the specific input that makes your verifier wrong.

The turn is not a quirk of that one verifier. Scaling Laws for Reward Model Overoptimization (Gao, Schulman and Hilton, 2023) fits the shape:

Two separate results, not one product: the KL cost of best-of-n on the first line, the fitted reward curve on the second. Best-of-N is optimisation pressure with a dial on it. Each increase in N costs a fixed, computable amount of KL divergence from the base policy — and when you optimise against a proxy reward model, the paper observes the gold reward first increasing and later decreasing, while the proxy's own score keeps climbing the whole time.

Which gives the sentence to keep. A reward model does not have random errors; it has a shape. Best-of-N walks straight along that shape, because it is literally selecting the sample the model scored highest. So the traces your loop keeps are, by construction, the most over-optimised traces you generated — the ones sitting furthest inside the verifier's blind spot. Then you fine-tune on them. This is how reward hacking enters a self-improvement loop: not through the training objective, through the selector.

Behavioural clustering: vote on what the program does

For code there is no final token to compare, so AlphaCode (Li et al., 2022) changed what "the same answer" means. Two programs are the same if they behave the same.

All but ten of a million programs are discarded by the selector, not the generator. The submission limit is the contest's; the funnel above it is the entire engineering problem.

Why largest-cluster-first works is the whole idea in one line, and the paper says it plainly: “there are many ways solutions can be incorrect while correct solutions tend to behave the same and therefore are grouped into larger clusters.” Majority vote, moved out of answer space and into behaviour space, where an agent's output actually lives.

It carried the system. AlphaCode's 41B model solved 34.2% of held-out CodeContests validation problems at ten submissions drawn from up to a million samples per problem, and in simulated evaluations on recent Codeforces competitions with more than 5,000 participants it averaged a ranking in the top 54.3%.

Where it breaks is the probes. The clustering is only as sharp as the generated test inputs: if no probe input distinguishes a correct program from a subtly wrong one, the two land in the same cluster and the vote counts them together. Behavioural equivalence is equivalence on the inputs you tried. For an agent, this is the directly useful version of the idea — run N trajectories, hash the end state each one leaves behind (files written, rows changed, calls made) and cluster on that. You never need the right answer. You need an equivalence relation and something to execute against.

Every selector above is a verifier in disguise. Majority vote verifies by agreement, ranking verifies by a learned score, clustering verifies by behaviour. Once you see that, the question sharpens into something you can design against: what are you checking, and at which point in the trace?

Outcome supervision, and the hole in it

The cheap answer is: check the last line. An outcome reward model is trained on one bit per solution — did it end correctly? Wherever you have an answer key, a test suite or a passing build, that bit is free.

The hole is structural. A right answer reached through a broken step passes. Let's Verify Step by Step (Lightman et al., 2023) builds on a known result — models trained with outcome supervision regularly use incorrect reasoning to reach the correct final answer (Zelikman et al., 2022; Creswell et al., 2022) — and names the consequence itself: automatic grading is not perfectly reliable, so false positives that reach the correct answer with incorrect reasoning are misgraded in the reward model's own training data.

Here is what that looks like in an agent trace. The agent is asked for the 90th-percentile latency. It sorts the samples and indexes int(0.9 * n) where it should use int(0.9 * (n - 1)). On the 101-sample array in the test, both expressions give index 90. The number is right. The code is wrong. The outcome check sees a pass and the trace is marked good — and on a 1,000-sample array the same code reads index 900 where it should read 899. For a one-shot answer that is a cosmetic problem. For a weight update it is a lesson: you have just told the model that this indexing is what success looks like.

A checker that reads only the last line cannot see any of that. It is not being careless. It was never shown the part where the mistake happened, so the only honest thing it can report is that the ending looked right.

Process supervision, and what it costs

The alternative is to label every step. Lightman et al. compare the two under a clean condition: solutions from a generator finetuned from the base GPT-4 model, best-of-N over 1,860 samples per problem, scored on 500 MATH test problems drawn uniformly at random.

Best-of-1,860 on 500 MATH problems, same generator throughout. The 5.8-point gap between the two reward models is what checking the middle of the trace is worth.

Step-level supervision buys two things. It localises the error, so the verifier's credit assignment matches the thing you would actually fix — which in a self-improvement loop means you can keep the good prefix of a bad trajectory instead of discarding the whole run. And it is harder to fool: a candidate now has to look right at every position rather than only at the last one, which raises the cost of exactly the adversarial solutions that ate Cobbe's 6B-verifier gains past 400 samples.

What it costs is people. The same paper released PRM800K — 800,000 step-level human labels across 75,000 solutions to 12,000 problems. That is about eleven human judgments per solution, on solutions a human has to read and follow line by line. Nobody is annotating production agent traces at that ratio.

Which is the real objection to labelling every step, and it is not a technical one. The method works. You have to decide whether the thing you are checking is worth that much of somebody’s attention, every week, forever — and for most teams the answer is no, which is why the next approach exists.

Buying step labels without buying labellers

Math-Shepherd (Wang et al., ACL 2024) removes the humans. The trick is to define a step's quality as its potential to reach the correct final answer: take the solution prefix up to that step, decode several continuations from it with a finetuned model, and score the step by how many of those continuations land on the gold answer. Steps that lead somewhere good get good labels. No annotator ever sees the problem.

It works well enough to train on. With step-by-step reinforcement learning against Math-Shepherd, Mistral-7B goes from 77.9% to 84.1% on GSM8K and from 28.6% to 33.0% on MATH; adding Math-Shepherd as a verifier on top pushes the two to 89.1% and 43.5%.

Note the catch, which follows from the definition rather than from any experiment: the label is the outcome, spread across positions. A step that happens to lead to the right answer scores well even when its reasoning is wrong — including the percentile bug above, whose continuations all reach the right number on the test array. Automatic process labels are far better than nothing and structurally closer to outcome supervision than the name suggests. They buy you localisation. They do not buy you a new source of truth.

Ensembles: several weak verifiers, weighted

In practice you never have one good verifier. You have four bad ones — a reward model, an LLM judge, a test run, a self-consistency score — that disagree, emit scores on incomparable scales, and come with no labelled data to weight them by.

Weaver (Saad-Falcon et al., 2025) attacks exactly that. It uses weak supervision to estimate each verifier's accuracy from the verifiers' pattern of agreement rather than from labels, normalises their output scales using dataset statistics, drops the ones carrying no signal, and combines the rest into a single weighted score. Weighted beats unweighted clearly — the verifiers differ too much for a plain average to be right. With Llama 3.3 70B Instruct as the generator and an ensemble of 70B-or-smaller judge and reward models, selection reaches 87.7% average accuracy across their reasoning and maths tasks: a jump comparable to the one between GPT-4o and o3-mini (69.0% to 86.7%) that otherwise took extensive finetuning and post-training.

The practical reading for a loop: you do not need a great verifier before you start. You need several cheap ones whose errors are not identical, and an honest estimate of how much to trust each.

A judge nobody judges

Here is the failure that will actually bite you, and it stops being subtle the moment you look for it. You added a verifier to catch the model's mistakes. Nothing is catching the verifier's.

The judge is much worse at this than the word suggests. JudgeBench (Tan et al., ICLR 2025) built 350 response pairs across knowledge, reasoning, maths and coding — responses generated by GPT-4o, labelled by objective correctness rather than by human preference, so a judge cannot win by picking the nicer-sounding answer. GPT-4o judging with a vanilla prompt scores 50.86% overall: 44.16% on knowledge, 47.96% on reasoning, 66.07% on maths, 61.90% on coding. On two of the four domains it is below a coin flip. An Arena-Hard-style prompt lifts the overall to 56.57%. The same model that wrote the responses cannot reliably tell which of two is correct.

And a judge that is genuinely good still invents things. OpenAI's critic models (McAleese et al., 2024) are trained with RLHF to write natural-language critiques of model-written code, and they are good: their critiques are preferred over human critiques on 63% of code containing naturally occurring model errors, and they catch more bugs than human contractors paid for code review. The paper's own caveat is the important half. The critics also raise objections against code that is fine, and the paper warns those invented bugs can push a reviewer into an error they would not otherwise have made; pairing a human with the critic hallucinates less than the critic working alone. A verifier that fabricates flaws is not neutral noise. It rejects correct traces at a steady rate, and its objections read rigorous, which is precisely why they survive review.

And no amount of compute fixes it. Stroebl, Kapoor and Narayanan (2024) give the ceiling: sampling again does nothing to the chance that a wrong answer slips past an imperfect verifier, so that chance caps how accurate a resampling loop can get, whatever compute you spend. On HumanEval and MBPP — whose unit tests have limited coverage — they find a strong correlation between a model's single-sample accuracy and its false-positive rate, and report that the optimal number of sampling attempts is often fewer than ten, because the harm from false positives bends the inference-scaling curve back down. Sampling harder does not out-run a broken judge. It finds the judge's holes faster.

The block below is that ceiling in arithmetic. Its two inputs — an agent right 20% of the time, a verifier that waves through 8% of the wrong answers — are stand-ins chosen to make the shape visible, not measurements off any real system. Put your own two numbers in and the conclusion does not move, because the line that computes precision has no sample budget in it.

Meta-verification: make the objection re-executable

You cannot fix this by adding a second judge with the same blind spots. You fix it by changing what a verdict is allowed to be. A verdict must carry evidence that something other than the verifier can reproduce.

Concretely: the verifier does not get to say "step 4 is wrong". It must name the step, quote the claim, and state the value it asserts is incorrect. A separate pass then re-executes exactly that — runs the expression, re-runs the named test, re-reads the cited line — and if the cited evidence does not reproduce, the objection is dropped and so is the verdict resting on it.

The move is the one behind Chain-of-Verification (Dhuliawala et al., 2023): draft a response, plan verification questions about it, answer those questions independently so the answers are not biased by the draft, then regenerate. On list-based Wikidata questions with Llama 65B that took precision from 0.17 to 0.36, and on longform biographies it took FactScore from 55.9 to 71.4 — past ChatGPT's 58.7 on the same task. Point the same pattern at the verifier instead of the generator and "I don't think step 4 is right" becomes "step 4 claims 18 × 7 = 136", which either re-executes or does not.

Independence is the load-bearing part. If the meta-verifier is the same model on the same prompt in the same context, it agrees with the objection because it wrote the objection. Execution is the cheapest independence there is, which is why verifiable rewards are worth reaching for wherever a task admits them: a compiler, a test suite, a type checker and a units check are all judges that cannot be talked round.

Your verifier is production software

Last rail, and the one teams skip. The verifier is a deployed component with a prompt, a model version and a threshold, and all three drift. Treat it like code: keep a frozen regression set of traces you know are good, traces you know are bad, and every specific failure the verifier has waved through before — then run it on every change to any of the three, and block the change on a regression. Agent evals covers how to build that set so it means something; evals in CI covers how to make it stop a merge.

Everything above rests on one word: filtered. Something has to say yes or no about each trace. The obvious answer is to train a scorer and ask it — and a trained scorer is a real answer, with its own literature (Let's Verify Step by Step is the one to read). It is also the expensive answer. You need labels to build it, it has to be maintained, and it can be wrong in ways nothing in your pipeline will tell you about.

But none of the three loops below needed a human-labelled correctness model. Each took the verdict from somewhere else: from the environment, from a test suite, and from the model reading its own output against a written rule. Those three are not interchangeable, and the difference between them is not quality. It is independence — whether the thing doing the checking can be wrong in the same way as the thing being checked.

Not a formula you can compute — a way to rank your options before you pick one. A test suite is high on independence and bounded on coverage. A self-critique is the reverse: it reads everything and catches little.

1. The environment answers

Most agent scaffolds pick a side. Chain-of-thought reasons but never touches anything, so it cannot notice it is wrong. An action-only policy touches things but has nowhere to put a plan. ReAct (Yao et al., 2022) interleaves the two in a single token stream: a thought, then an action, then whatever the environment returns, then the next thought — which can read that return. The thought is a move in an augmented action space that changes nothing outside the context window. That is exactly its job. It is the place where an observation becomes a plan.

The numbers say how much was sitting in the environment unread. On ALFWorld — 134 unseen household games, PaLM-540B — ReAct's best of six prompts succeeds on 71% of games, and its average over those six is 57%. The same model acting without thoughts gets 45% on its best of six. BUTLER, an imitation learner trained on roughly 100,000 expert trajectories per task type, gets 37% on its best of eight. On WebShop's 500 test instructions, ReAct scores 66.6 with a 40.0% success rate, against 59.9 / 29.1% for imitation learning on 1,012 human trajectories and 62.4 / 28.7% for imitation plus RL, which adds 10,587 training instructions on top of those. ReAct was prompted with one or two in-context examples.

A search that returns nothing, a click that lands on the wrong page, a traceback: these are facts, produced by something with no opinion about your model. They cost one tool call and they arrive whether or not anyone is grading. The honest half of the same table is the human row on WebShop — 82.1 with 59.6% success. Reading the observation is not the same as knowing what to do with it. The pattern itself and the wiring around it each have their own page; what matters here is that the loop produces traces whose correctness the world already judged.

2. The tests answer

The strongest free signal in software is the one your CI already runs. A test passes or it does not. The verdict costs a subprocess, it is reproducible, and it does not care what the model believes about its own work. This is the ground truth that verifiable rewards are built on, and if you already run tests in CI, you already own a labelling machine.

There are two ways to spend it. Filter at inference is AlphaCode: generate up to a million programs for one problem, run them against the example tests printed in the problem statement, and watch roughly 99% of them die. What survives goes to a submission budget of ten. That system solved 34.2% of held-out CodeContests validation problems at 10@1M and placed in the top 54.3% averaged over ten real Codeforces contests with more than 5,000 entrants each.

Train on it is RLEF (Gehring et al., 2024). The episode is a conversation. The model writes a program; the public tests run; if they fail, the failure text is pasted back into the conversation and the model writes again — up to three turns. The reward is paid at the end over the whole suite, public and private — and it is the private half, which the model never saw, that makes it a real gate: +1 if every test passes, −1 if any fails, −0.2 for a turn that produced no runnable program. The optimizer is PPO. On the CodeContests test set at 1@3, Llama 3.1 8B Instruct goes from 10.5% to 16.0% and 70B Instruct from 27.5% to 40.1%; at 10@100 the 8B goes 24.8% → 28.7% and the 70B 50.3% → 54.5%. Stated as sample count instead of accuracy: on that test set the RLEF'd 70B clears AlphaCodium-on-GPT-4, the standing record at the time, from a single rollout, where that record needed five submissions drawn from a hundred samples — though on the validation split the older system still leads it, 44 to 37.5.

The public/private split is the design, not a detail. The tests the model can read mid-episode are the ones it is allowed to iterate against. The verdict that pays the reward also runs the ones it cannot see. Collapse the two and you are no longer training a model to write correct programs — you are training it to special-case the tests it was shown.

3. The model answers, against a written rule

Constitutional AI (Bai et al., 2022) is the third shape. Sample a response to a red-team prompt. Ask the same model to critique that response against a principle drawn at random from a written list — sixteen of them, on harmlessness — and then to revise it. Repeat. Fine-tune the original model on the revisions. Then run RLAIF: the model picks which of two responses is better, a preference model is trained on those AI-made comparisons, and reinforcement learning optimizes against it. The list of principles is the only human oversight of harmlessness in that loop, and only of harmlessness. Human labour is still in three other places: the model being critiqued is already a helpful RLHF model, the revisions are fine-tuned in alongside 135,296 human-written helpfulness prompts, and the preference model the paper ends up with is an explicit hybrid — human-labelled helpfulness comparisons next to the AI-labelled harmlessness ones.

It works. The paper reports harmlessness improving monotonically with the number of revisions, and notes that critiquing before revising helps small models more than large ones, which are nearly as good revising directly. The stated goal is worth repeating because it is unusual: harmless but non-evasive — an assistant that engages with a harmful request by explaining its objection, rather than one that has learned to say nothing.

This is a model grading a model. That it works here is a fact about the task, not a fact about self-critique.

The number that separates them

Huang et al., ICLR 2024 — Large Language Models Cannot Self-Correct Reasoning Yet — ran the loop everyone runs: answer, review your answer, answer again. The only thing they varied is who decides when to stop.

Give the loop an oracle label — a real verifier saying "that's right, stop" — and GPT-3.5-Turbo on GSM8K's full 1,319-problem test set goes from 75.9% to 84.3%, and on CommonsenseQA's 1,221-question dev set from 75.8% to 89.7%. Take the oracle away and let the model judge its own answer, same model, same questions, same prompts, and it goes 75.9% → 75.1% → 74.7% across two rounds — for five model calls instead of one. CommonsenseQA falls off a cliff: 75.8% → 38.1% after one round, clawing back only to 41.8% after the second. GPT-4 on GSM8K drops 95.5% → 91.5% → 89.0%, though on a 200-question random sample rather than the full test set the GPT-3.5 rows use. On GSM8K, GPT-3.5 left 74.7% of its answers untouched; among the ones it did change, it turned more right answers wrong than wrong answers right. The same paper checks the multi-agent version and finds that several copies of a model critiquing each other does no better than self-consistency once you hold the number of responses equal.

Nothing there is about capability. The model knew exactly as much in both columns. What changed is where the stopping verdict came from.

Which is why "external good, internal bad" is the wrong summary. The real question is whether the check is a different question from the one that produced the answer. Constitutional AI's critique asks something close to a lookup — does this response contain a slur, does it explain its objection — and the model can answer it without redoing the task. A reasoning self-critique asks the model to find an error using the same machinery that made it. One is a second question. The other is the first question, asked twice, at five times the cost.

if trace.correct is one line, and it is four decisions short. Correctness tells you a trace is eligible. Which eligible traces actually go in the batch is a separate question, and it is the one that decides what the loop becomes — because that batch is the curriculum for the next version of the model, and that version generates the batch after it. Get this wrong and nothing breaks loudly. The loop just narrows.

Four predicates and a budget. The order matters: cut to the budget last, or you spend it on whichever traces happened to arrive first.

1. Correctness — and what it is allowed to mean

The section above is about where this verdict comes from. The decision left over is narrower: does your gate check the answer or the route? An answer-level check admits every trace that landed on the right result by a wrong path, and on multiple choice that has a floor built into the task. STaR's authors call it out for CommonsenseQA — five options means roughly a fifth of attempts hit the right answer no matter what the reasoning did, and the filter cannot tell those apart from reasoning that worked. Answer-checking is cheap, and cheap here has a number attached to it.

2. Near-duplicate rejection

A hundred copies of one solution is one solution, charged at a hundred times the budget. That is the polite version. The real cost is that the optimizer sees that token sequence a hundred times, and the model you get back is the one that likes it. Diversity is not a nice property of a training set; it is the only thing distinguishing a training set from a prior.

Self-Instruct (Wang et al., 2022) is the clearest published setting. Starting from 175 hand-written seed tasks and vanilla GPT-3 (davinci), it grew 52,445 instructions covering 82,439 instances — and a newly generated instruction entered the pool only if its ROUGE-L similarity against every instruction already in the pool was below 0.7. The other filters are just as blunt: drop instructions that mention things a text model cannot do (image, picture, graph), drop instances identical to one already kept, drop instances with the same input but a different output. Fine-tuned on its own filtered output, GPT-3 went from 6.8 to 39.9 ROUGE-L on the unseen tasks of SuperNI — 0.9 behind InstructGPT-001's 40.8, without the private user data. The generation side of that pipeline is its own subject; the filter is the part that belongs here.

The threshold is the interesting number. 0.7 is not a constant of nature. It is the point at which "different enough to be worth a training slot" was true for that pool, and it moves with what the pool already contains.

For reasoning traces, RFT (Yuan et al., 2023) dedupes on structure instead of surface. Sample k = 100 reasoning paths per GSM8K question at temperature 0.7, keep the ones that reach the right answer, extract the list of equations each path used, and keep exactly one path per distinct equation list — with ordering counted, so 3+4=7 and 4+3=7 are different lists. LLaMA-7B: 35.9% with ordinary supervised fine-tuning, 41.7% after RFT at k=100, 49.3% for RFT-U33B, which pools samples across several models. The relationship the paper states is the one to carry away: performance rises with the number of distinct reasoning paths, not with the number of correct ones.

The strongest form dedupes on behaviour. AlphaCode trains a separate model to invent new test inputs, runs every surviving program on them, and groups programs that produce identical outputs into clusters. Submissions are drawn one per cluster, largest cluster first — the reasoning being that there are many ways to be wrong and comparatively few ways to be right, so correct programs pile into the big clusters. Two programs sharing no tokens land together if they do the same thing. No string filter can do that.

3. Difficulty banding

An easy problem yields many correct traces. A hard one yields few, or none. Keep everything that passed and your training set is a picture of which problems were easy — which is the one thing the model already knew.

ReST-EM (Singh et al., 2023) puts a number on the fix. On MATH's 7,500 training problems it draws 32 samples per problem; on APPS Introductory's 2,342 problems, 64. Then a hard cut-off: at most 10 kept solutions per problem, on both datasets — a cut-off the paper credits to STaR. And the cost of skipping it shows up in the iteration curve. On APPS almost all of the gain arrives in the first round, and running further rounds regressed performance on APPS and on HumanEval. A small problem set, resampled, gets memorized.

Reinforcement learning states the same rule exactly rather than heuristically. Under GRPO with outcome supervision, a sample's advantage is its reward minus the mean reward of its group, divided by that group's standard deviation. A prompt where every attempt succeeds has zero spread. So does one where every attempt fails. Either way every advantage in the group is zero and the prompt contributes nothing to the update, however many samples you spent generating it. The band that pays is the one where the pass rate sits strictly between 0 and 1.

So banding is not hygiene. In the RL case it is the difference between spending compute and burning it; in the fine-tuning case it is the difference between a curriculum and a histogram.

4. Batch — how many, how often

Two knobs pulling against each other. Within a round, budget is better spent on more distinct problems than on more solutions to problems you have already solved — which is the per-problem cap again, seen from the other side. Across rounds, every round's data is produced by the model the previous round made, so the distribution moves underneath you; ReST-EM's second-iteration regression is what that looks like when the problem set is too small to move with it.

A defensible execution order, which is not the order these are best explained in: filter for correctness, band by difficulty (cheap, kills whole problems at once), dedupe within each surviving problem, cap per problem, then cut to the round budget. And write the numbers down. The batch you hand to fine-tuning is defined by four thresholds somebody chose, and if they are not versioned alongside the checkpoint, nobody can explain the run six weeks later.

The fifth question: near-misses

Everything above throws failures away, and that is a problem, because the problems a model fails are the ones worth learning. STaR hits the wall directly: the loop stops picking up new training problems, because a problem it never solves never produces a trace, and a problem with no trace produces no gradient. The filter has quietly capped the loop at what the model could already do on day one.

Their answer is rationalization. Take a problem the model got wrong. Hand it the correct answer as a hint and ask for a rationale in the same style. If it now produces reasoning that arrives at that answer, keep the rationale — with the hint deleted, as though the model had reasoned there unaided.

The numbers, all GPT-J at 6B. CommonsenseQA dev: 20.9% few-shot direct, 36.6% few-shot chain-of-thought, 60.0% for GPT-J fine-tuned on the whole training set to predict answers directly, 68.8% for STaR, 72.5% for STaR with rationalization — against 73.0% for a directly fine-tuned GPT-3, a model 30× larger. GSM8K test: 3.0% few-shot direct, 3.1% few-shot CoT, 5.8% fine-tuned direct, 10.1% STaR, 10.7% with rationalization.

The column beside accuracy is the one that explains the mechanism: how much of the training set the loop ever managed to touch. On CommonsenseQA the final model trained on 86.7% of the training problems — 78.2 points from ordinary generation, and 8.5 points that only rationalization reached. On GSM8K the total is 28.7% with just 0.5 points from rationalization. Same technique, an order of magnitude apart in yield, because on a five-choice question a backward rationale usually lands on the given answer and on a word problem it usually does not.

On arithmetic the effect is starkest. Two-digit addition sat under 1% few-shot and reached 32% after a single fine-tuning iteration on the model's own generated scratchpads. What the paper puts down to rationalization is the shape of the curve, not that jump: without it the model improves stage-wise, staying poor on n-digit sums until it is good at n−1, while with it several lengths come in at once. Run out to 16 iterations, STaR on arithmetic reaches 89.5% overall accuracy across 1–5 digit sums, against 76.3% for a baseline trained on 10,000 examples with no rationales at all.

Now the part to be careful about. A rationalized trace is reasoning produced by a model that already knew the answer. It can be a plausible path that does not actually derive the result — work that looks like work. Keep them, because the alternative is never touching the hard tail. But keep them in their own band, keep the count small against the generated ones (STaR's best case was 8.5 points inside 86.7), and never let a rationalized trace be the only evidence that a class of problem is solved.

“Fold it back in” is the vague half of the loop. There are six places a lesson can actually go, and they are not variations on each other — they differ by orders of magnitude in what they cost, and by everything in what happens when the lesson turns out to be wrong. Same lesson, six addresses:

the context you assemble for each request, an external memory the agent reads and writes, the tool set it can call, the scaffold that decides how many times it thinks, the verifier that decides what counted as success, and the weights.

Choosing between them is the whole job. Everyone arrives at this page wanting the sixth one, because training is what improvement looks like in papers. Most production improvement lands in the first five, and the cases where that is the wrong call are narrower than they look.

1 · The context

Everything you put in front of the model this request: the system prompt, the instructions, the few-shot examples, the retrieved snippets. The improvement is a text edit, it takes effect on the very next request, and it is the substrate people underrate hardest — because prompt engineering sounds like fiddling.

Measured, it is not fiddling. In Large Language Models as Optimizers, Google ran an LLM as the optimizer over the instruction itself: propose candidate instructions, score each on 3.5% of the GSM8K training set, feed the scored history back as the next prompt, repeat. The instruction it converged on for PaLM 2-L — Take a deep breath and work on this problem step-by-step — scored 80.2% on GSM8K, against 71.8% for the standard Let's think step by step and 34.0% for an empty instruction. Identical weights. More than eight points from one sentence.

Read the condition, though, because it is the lesson: that sentence was tuned against that scorer model. An instruction optimized on one model is a fact about that model, not about language models. Context improvements are cheap to make and cheap to invalidate — the next base model resets them. Context engineering covers how to assemble the window; here, the only property that matters is that this substrate is free to change and free to abandon.

2 · External memory

A store the agent writes to when something goes well or badly, and reads from when a similar situation returns. Reflexion is the pure form: after a failed attempt the agent writes a verbal post-mortem into an episodic buffer and retries with that text in context — no gradient anywhere — reaching 91% pass@1 on HumanEval with GPT-4 underneath, against the 80% GPT-4 baseline the paper compares to.

Agent Workflow Memory pushes the same substrate further: instead of storing what went wrong, it induces reusable workflows — a description plus a sequence of state, reasoning and action — out of past traces. With GPT-4, that took WebArena success from 23.5% to 35.5% and Mind2Web cross-task step success from 36.2% to 45.1%. The variant worth staring at is the online one: it induces workflows from its own successful runs, judged by an evaluator, with no human supervision. That is a complete self-improvement loop whose substrate is memory. Nothing was trained.

Memory's defining property is not speed, it is scope. A bad memory only damages the requests whose retrieval hits it. That is why memory is the correct first home for anything you are not yet sure about — see agent memory for how the store and the retriever are actually built.

3 · The tool set

A tool is a memory with a type signature. Voyager is the canonical demonstration: a GPT-4 agent in Minecraft that writes new skills as executable code, verifies each by running it, stores it keyed for retrieval, and composes stored skills into harder ones. What that library is worth shows up in the play: measured against the prior state of the art, the agent ended up with 3.3× the distinct items, ranged 2.3× as far, and reached key tech-tree milestones as much as 15.3× sooner.

The property that matters for a loop is not that the skill library grows. It is that a stored skill is executable, so it can be tested before it is trusted. You cannot unit-test a remembered paragraph of advice. You can unit-test a function. Every substrate above this one is judged by a model; this one can be judged by a compiler.

4 · The scaffold

The control flow around the model: how many samples you draw, in what order, with what search, with what retries, with what self-check. Self-consistency is the one-line version — sample 40 reasoning paths instead of one and take the majority answer, which moved PaLM-540B on GSM8K from 56.5% to 74.4%. Tree of Thoughts is the elaborate version: on Game of 24, GPT-4 with chain-of-thought solved 4% of instances and the same model inside a search over thoughts solved 74%.

In both, the weights never moved. What moved was how much thinking the harness bought. Which is also the catch: that cost is paid per request, forever. Forty samples is forty times the tokens, on every call, for as long as the feature ships. Scaffold improvement is improvement you rent — see test-time compute for the pricing, and loop engineering for how the harness is structured.

5 · The verifier

The thing that decides what counted as a success — and therefore what all five other substrates are permitted to learn. It is the highest-leverage substrate on this list and the one teams skip, because improving it means paying for labels.

Let's Verify Step by Step priced that exactly. Sampling 1860 solutions per problem on a 500-problem subset of the MATH test set, with GPT-4-derived models, ranking by a process-supervised reward model solved 78.2%; the same setup with an outcome-supervised model solved 72.4%, and plain majority voting solved 69.6%. The difference between grading the answer and grading each step was worth almost six points at fixed sampling budget. The price of it was PRM800K: 800,000 step-level human labels.

A verifier improvement propagates into every other substrate at once, because every other substrate is filtered through it. It is also the only substrate with no independent check on it — everything else is graded by the verifier, and the verifier is graded by you. Agent evals and verifiable rewards are both, read this way, work on the same object.

6 · The weights

The substrate that sells the one thing none of the others can: transfer to tasks you have never seen. STaR took GPT-J, 6B parameters, from 5.8% on GSM8K when fine-tuned to answer directly to 10.1% by fine-tuning on its own filtered correct rationales, and to 10.7% once failed problems are retried with the answer supplied as a hint. GRPO took DeepSeekMath-Instruct 7B from 82.9% to 88.2% on GSM8K and 46.8% to 51.7% on MATH, chain-of-thought only, no tools and no voting.

Both are improvements on held-out test problems — though from the same two benchmarks the training questions were drawn from, so they measure a better policy on a familiar distribution rather than transfer to a task family the model has never met. Transfer is the argument for this substrate. These two numbers are not the evidence for it; they are the evidence that the mechanism works.

The cost floor is lower than it used to be — LoRA reduces trainable parameters by 10,000× against full fine-tuning of GPT-3 175B with Adam, and GPU memory by 3×, with no added inference latency. But even at that discount the unit of work is a training run, an eval sweep, a canary, and a rollout, measured in days. Fine-tuning covers the mechanics.

The six, side by side

The page's two-column memory-versus-weights table, extended. Read it down the reversibility column first, then down the blast-radius column, and notice they do not agree.

the table scrolls sideways →

Two things that column pair tells you

First, weights are not irreversible — they are unsubtractable. You keep checkpoints; you can always roll back. What you cannot do is remove one lesson. Rolling back returns the model to the state before the update, which also discards everything good that was in that batch. Memory supports deletion. Weights support only rewind. That is a sharper and more useful statement than “hard to undo”, because it tells you what the recovery procedure actually costs: a batch, not a bad row.

Second, reversibility and blast radius are different axes, and collapsing them is what puts teams in incidents. A prompt edit is perfectly reversible and completely global — it touches every request from the second you deploy, so if it is wrong you find out at full traffic. A weight update is the opposite: the hardest thing on the list to take back, and the easiest thing on the list to release slowly, because a checkpoint is an artifact you can serve to 1% of traffic and watch. The reversible substrates are precisely the ones that tempt you to skip the staged rollout. That is how a one-line edit becomes a site-wide regression while the training pipeline, which everybody was nervous about, ships fine.

The two ways to get it wrong

Landing a specific fact in the weights: you spend a training run, you accept permanence, and you buy generalization for something that will never recur in a different form. One vendor's date format does not have a task family. It has a row.

Landing a general lesson in memory: you keep paying retrieval for something the model should simply know, you get nothing on the tasks you have not seen yet — which is the population you actually care about — and your context fills with near-duplicate entries that a single weight update would have compressed. If your memory store has two hundred rows that all say the same thing in different words, the store is telling you it is the wrong substrate.

Everything so far has been one pass: improve something, ship it, look at the result. Closing the loop means the output of the run becomes the input to the next update, automatically, on a cadence. The simplest version of that is small enough to hold in your head.

STaR's loop. The arrow at the end returns to the start with the improved model, so the next round's samples are drawn from a better policy than the last.

STaR: the smallest complete loop

STaR ran exactly that on GPT-J, a 6B model, and the numbers are worth keeping because they are so unglamorous. On GSM8K: few-shot chain-of-thought scored 3.1%, fine-tuning the model to predict the answer directly scored 5.8%, and the loop — sample, keep correct, refit, repeat — scored 10.1%. On CommonsenseQA the same 6B model reached 68.8% from the plain loop and 72.5% once failed problems were retried with the answer as a hint, against 73.0% for a fine-tuned GPT-3 with 175B parameters. A model thirty times smaller, within half a point, on its own output and a nudge.

STaR adds one move worth naming. When the model fails a problem, it is handed the correct answer as a hint and asked to produce a rationale for it — reasoning backwards from a known destination is much easier than finding it — and the rationale it produces is kept if it lands on the answer. That is rationalization, and on GSM8K it moved 10.1% to 10.7%. Be clear-eyed about what it manufactures: a hinted rationale is a reconstruction of a path to an answer the model was given, not a record of how it would have found it. You are training on plausible derivations. Sometimes that is exactly what you want, and sometimes it is how a model learns to produce confident justifications for conclusions it did not reach.

Why the filter is the gradient

STaR looks like data curation. The paper shows it is not. Write the model as sampling a rationale before the answer, take the gradient of the objective, and an indicator appears in front of every term:

The indicator is 1 when the attempt reached the right answer and 0 when it did not, so every failed attempt multiplies by zero. Keeping only correct traces is not a preprocessing choice — it is what this objective's gradient already does.

STaR then takes two liberties on top, both of which the paper names: it decodes greedily rather than sampling, which cuts variance at the cost of biased exploration, and it takes several gradient steps on the same batch, which is what policy-gradient implementations do anyway. So supervised fine-tuning on filtered correct traces is not like reinforcement learning. It is a particular, blunt reinforcement learning algorithm — one whose advantage function can only be 0 or 1.

What changes when the update is a policy gradient

Once you see the indicator, you can see what it throws away. A failed attempt cost you exactly as much compute as a successful one, and STaR learns nothing from it. Replace the indicator with a real-valued advantage and both signs do work: better-than-average attempts get pushed up, worse-than-average attempts get pushed down, and “worse than average” is information you already paid for.

That requires an average — a baseline. PPO learns one: a value network, typically the size of the policy, trained alongside it to predict the return from a partial sequence, with the policy update clipped by the probability ratio so no single batch can move the policy too far. It works, and it doubles your memory footprint and adds a second thing that can be wrong.

GRPO deletes the value network by noticing what the baseline was for. Sample a group of G answers to the same question, grade them all, and standardize each answer's reward within its own group:

The group mean is the baseline. It is an unbiased-enough estimate of “how well does this policy usually do on this question”, which is precisely what the value network was being trained to guess.

The DeepSeekMath run that introduced it is a useful sense of scale: roughly 144k chain-of-thought questions drawn from GSM8K and MATH, G = 64 samples per question, policy learning rate 1e-6, KL coefficient 0.04. That is the shape of the bill — sixty-four generations per question, before a single gradient step.

The same machinery with purely rule-based rewards is what produced DeepSeek-R1-Zero: starting from a base model with no supervised reasoning data at all, AIME 2024 pass@1 went from 15.6% to 71.0%, and to 86.7% with majority voting over 64 samples. The reward was two rules — is the final answer right, and is the output in the required format. Hold on to that second rule; it comes back below.

And if what you have is preference pairs rather than a checkable answer, DPO skips the loop entirely: it reparameterizes the reward so that the optimal policy has a closed form, which collapses the whole RLHF problem into a classification loss over the pairs — no reward model, no sampling from the model during training. What you give up is the online part. The model never sees its own fresh mistakes, so DPO improves a policy but does not close a loop.

What breaks in long-chain training

Group-relative advantage has a failure mode you meet immediately at scale, and it follows directly from the formula above: if all G samples for a question get the same reward, every advantage in that group is zero. The question consumed 64 generations and contributed no gradient. As the model gets better, more questions become all-correct, so the fraction of your batch that teaches anything shrinks while the bill stays flat. The training curve flattens and the invoice does not.

DAPO — which reached 50 points on AIME 2024 from a Qwen2.5-32B base, above DeepSeek-R1-Zero-Qwen-32B's 47, in half the training steps — is essentially a list of these, each with its fix:

the table scrolls sideways →

There is a subtler one still. A critical reading of R1-style training found an optimization bias inside GRPO itself: the update quietly pays for length, and pays for it most on the answers that were wrong. The unbiased version, which the authors call Dr. GRPO, is one ingredient in a minimalist recipe that reached 43.3% on AIME 2024 from a Qwen2.5-Math-7B base. The general shape is worth internalizing: in a long-chain loop, any term that scales with sequence length becomes a reward for length, whether or not you meant it to be one.

Reward hacking, which is not an edge case

Every loop on this page optimizes a number you wrote down. That number is a proxy for what you want, and the gap between the two is not a bug to be patched — it is a space, and the optimizer's job is to search spaces.

The systematic study is Pan, Bhatia and Steinhardt's work on reward misspecification: four RL environments with deliberately imperfect rewards, and then agents of increasing capability — more model capacity, finer action resolution, longer training. The finding is the uncomfortable one. More capable agents often exploited the misspecification, scoring higher on the proxy reward and lower on the true reward than less capable ones. Worse, some of those failures arrived as phase transitions: a capability threshold at which behaviour flips qualitatively and true reward drops sharply. The proxy curve you are watching on the dashboard stays smooth and rising straight through it.

For language models specifically, the clearest demonstration is the U-SOPHISTRY result: after RLHF, models got better at convincing human evaluators without getting better at the task. Evaluators working under a 3-to-10-minute time limit saw their false-positive rate rise 24.1% on the QuALITY question-answering task and 18.3% on the APPS programming task. Human approval went up. Correctness did not. The optimizer found the cheaper of the two. Note the condition, because it is the part that generalizes: this is what evaluation looks like on a clock, which is how most review in production actually happens.

This is the reasoning behind DeepSeek-R1's rule-based rewards, stated plainly in the paper: they declined to use a neural reward model for the large-scale run because it can be reward-hacked, and because retraining it costs more compute and complicates the pipeline. A checkable answer cannot be argued with. That is the entire argument for verifiable rewards, and it is why the design of the environment — what is measurable, what is scorable, what is gameable — is the real work of RL environment engineering.

Watch it happen

The block below adds a small format bonus to a correctness reward — exactly the second rule in R1-Zero's reward, the one that seems harmless — and lets a policy over twelve answer strategies find it. Reward climbs and settles at its ceiling. Expected accuracy falls by a fifth from where the policy started, before any updates. Nothing malfunctions at any point; the bonus is paid exactly as specified, on every answer.

The arithmetic is the whole lesson, and it is worth measuring strategy to strategy rather than from the starting policy: going from the most accurate of the twelve to the most formatted costs 0.22 of accuracy and pays 0.30 of bonus. Every individual trade is profitable under the reward you wrote. The optimizer takes all of them, in order, and lands on the worst strategy in the list.

Run the loop you have been building — generate, verify, filter, refit — six rounds in a row, and watch three numbers instead of one. Pass@1 climbs. Coverage rises once and then falls for the rest of the run. Answer entropy drops toward zero. That is not three findings. It is one chart read three ways. The chart is a simulation of that loop, not a training run: the shape is real and the scale is not. Every measured number further down this section is cited to the paper it came from.

Each round does the same four things, and every one of them is a dial an earlier section put in your hands:

1. Generate. Sample n attempts per task from the current policy. The spread of those attempts is the raw material for everything that follows.
2. Verify. Run the checker. It returns one bit per trace, and it is wrong some fraction of the time — in both directions.
3. Filter. Apply the policy you wrote in section 07: the accept threshold, the dedupe rule, the per-task cap, the decision about what counts as the same answer.
4. Refit. Train on what survived. Then return to step one and sample from the model that refit produced.

Step four is the whole difference between this and an evaluation harness. The next round's generator is the last round's product, so whatever the filter preferred is now what the sampler produces more of, which is what the filter sees more of. The loop is a feedback path and your filter is its transfer function. Nothing else in the system has that much leverage over where it ends up.

What follows is simulated. The six rounds below are this page's own model of the loop — twelve strategies and a filter you can change, not a policy gradient over a real model. Read the direction each of the three curves moves and where it turns; do not read the scale off it. The measured results start immediately after it, with Yue et al.

Why the three curves are one curve

Pass@1 is the probability that one sample is right. Coverage — pass@k at a large k — is the probability that at least one of k samples is right. Entropy is how wide the distribution you are drawing those samples from still is. Refitting on your own filtered output is a concentration operator: it moves probability mass onto whatever passed. Concentration raises the first number by construction and lowers the third by construction. The second is the only one that tells you what the trade cost, because a task that was only ever solved by an approach the model has stopped proposing quietly leaves the solvable set — and nothing in a pass@1 dashboard reports a departure.

The cleanest measurement of this is Yue et al.'s Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025), which ran pass@k out to large k on RLVR-trained models and the base models they came from, across the Qwen2.5 and LLaMA-3.1 families and math, coding and visual-reasoning benchmarks. At k = 1 the RLVR models win, which is the result everyone quotes. At large k the base models win: on Minerva with the 32B model, the base model beats the RL-trained one by roughly 9 points at k = 128. The sharpest number is the per-problem breakdown — on AIME24, 13.3% of problems were solved by the base model and not by the RLVR model, and 0.0% went the other way. Not one problem on that benchmark was newly unlocked — and on MATH500, the closest thing to a counterexample, the RLVR model's exclusive share was 1.0% against the base model's 3.6%. The training made the model faster at finding paths the base model already had, and it paid for that by making other paths unreachable.

The uncomfortable part is that nothing malfunctioned. The loop moved probability onto what passed, which is what you built it to do. Concentrating and narrowing are not two behaviours that happen to co-occur; they are one motion described twice.

Entropy turns the same story into something with a ceiling you can read off. Cui et al.'s The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025) fit a two-coefficient law relating downstream performance R to policy entropy H.

Two coefficients, fit across 11 base models from four families — Qwen2.5, Mistral, LLaMA and DeepSeek-Math, 0.5B to 32B — over more than 200 measured points (Cui et al., 2025). Read at H = 0, it gives the score the run converges to once the policy has no spread left to spend, which makes remaining headroom a function of unspent entropy rather than of remaining steps.

Their runs spent it fast. Seventy-three percent of the entropy was consumed and 76% of the performance gained in the first 200 gradient steps — one twelfth of training. The first third of training accounted for 94% of the entropy loss and 93% of the gain. Performance was not being created there. It was being traded from entropy, and the exchange was mostly finished before the loss curve looked interesting.

Read that as a budget rather than a curve. You begin with a fixed amount of exploration, the loop spends nearly all of it early, and whether you were watching at the time makes no difference to the bill.

You do not need RL to watch the saturation happen. Beyond Human Data (Singh et al., 2024) ran the plain generate-filter-finetune loop — ReST^EM — on PaLM 2-S, S* and L over MATH and APPS, and where it plots train against test — MATH with PaLM 2-L, APPS with PaLM 2-S* — training-set accuracy rose roughly linearly with each iteration while test accuracy did not: gains after the first iteration were small on MATH, and on APPS the second iteration was an outright regression. The loop kept learning. It stopped learning anything that transferred, after round one.

A verifier that is wrong a little is wrong more every round

A checker has two ways to be wrong, and they do not compound the same way. Keeping them apart is most of what separates a loop that holds from one that does not.

False positives — accepting a trace whose answer is wrong — put a bad target in the training set. One round of that is survivable: the set is mostly right and the gradient averages. The danger is structural rather than statistical. The policy is being optimized against the verifier, which means it is actively searching for the inputs the verifier scores wrongly. The Pitfalls of Rule- and Model-based Verifiers study (2025) caught this on camera: a model-based verifier with respectable static accuracy was used as the RL reward, and after roughly 450 training iterations the training reward diverged from the oracle reward while the policy learned to emit a single character such as {, or long runs of meaningless text, that the verifier scored as correct. The verifier never got worse. The policy got better at it. That failure mode lives in full at verifiable rewards.

False negatives — rejecting a trace whose answer is right — are the ones that compound quietly, because they are not randomly distributed. The same study measured a widely used rule-based math verifier at around 0.92 recall on completions from long-chain-of-thought models such as DeepSeek-R1-Distill-Qwen-7B and 32B: roughly one correct answer in twelve discarded, and discarded overwhelmingly for expressing the answer in a form the matcher did not anticipate — π/4 where it wanted 45°. Their stated trend is the uncomfortable one: the better the policy gets, the worse a rule-based verifier is at supervising it, because a more capable model writes more unusual correct solutions, and unusual is exactly what a pattern matcher cannot read. Your filter is not adding noise to the training set. It has a taste, and its taste is for the conventional.

Now put that taste in a loop. Round one credits unconventional-but-correct work at roughly 92% of its value, so the refit gives it proportionally less weight. Round two samples from that model, so there is less unconventional work to reject. Round three has less again. No single round does anything dramatic — which is the problem, because pass@1 is rising the entire time.

The block below runs exactly that, six rounds, with the filter's blind spot as the only variable. It is the reward, not the optimizer — a twelve-strategy softmax standing in for a policy gradient over billions of weights — so the shape of the three curves is real and the scale is not. Nothing here reproduces a training run. What it reproduces is the arithmetic of concentration.

What the model-collapse papers found, and what they did not

The general question — what happens to a model trained on generated text — has its own literature and its own handbook page. The canonical result is Shumailov et al., published as The Curse of Recursion in 2023 and as AI models collapse when trained on recursively generated data in Nature 631, 755–759 (2024). Their language-model setting is worth stating precisely, because almost every summary of it drops the part that matters.

They fine-tuned OPT-125m on wikitext2, generated from it with five-way beam search in 64-token blocks, built a synthetic dataset the same size as the original, trained the next generation on that, and repeated. The baseline model sits at 34 mean perplexity on real data. Each recursive generation degrades. By the later ones the samples have degenerated into repetition — the paper's own example wanders from cathedral architecture into a list of jackrabbits in assorted colours. Their account of the mechanism is the durable part: the rare events at the thin ends of the original distribution stop being sampled, and then stop being represented at all, driven first by statistical approximation error (a finite sample simply misses rare events, every generation, compounding), then by the model family's limited expressiveness and by the learning procedure itself.

Then comes the control that most citations omit. Gerstgrasser et al. (2024) ran the same recursive-fitting experiment with one change — instead of replacing the real data with each generation's synthetic data, accumulate: keep the original corpus and add the synthetic data to it — on their own setup, small transformers trained on TinyStories rather than Shumailov's OPT-125m on wikitext2. Collapse stops. At iteration four, GPT-2 (9M) sits at 1.74 cross-entropy accumulating against 2.39 replacing; Llama-2 (125M) at 1.59 against 2.23. They also prove it for linear regression: under accumulation, test error has a finite upper bound no matter how many iterations you run, while under replacement it grows without bound. Collapse is not a property of synthetic data. It is a property of throwing the real data away.

So: is an agent training on its own verified traces doing the thing that collapses? Related, but not identical, and the differences are all load-bearing.

The first three rows are why you should not simply transplant the collapse result onto your pipeline and panic. A verifier is an enormously strong prior that the collapse setting does not have; keeping the original SFT mixture puts you in the accumulate regime that provably does not diverge; and losing density over wrong answers is the entire point of the exercise.

The fourth row is why you should not relax either. Collapse is a loss of variance. Your loop is designed to remove variance. The only question separating the two is which variance it removes — and a filter whose false negatives land preferentially on unusual-but-correct work is, mechanically, running the collapse experiment on the part of the distribution you most needed to keep. That is the precise sense in which they are the same phenomenon: the same operation, with the sign decided entirely by the filter you wrote in section 07.

The dials, and which curve each one bends

And measure accordingly. Pass@1 every round is not a measurement, it is a formality — concentration guarantees it. Hold out a fixed task set, sample it with the same sampler settings every round, and record three things: pass@k at the largest k you can afford, the count of distinct passing solutions per task, and token-level entropy on a fixed prompt set. Agent evals covers building that harness. A round that raises pass@1 and lowers pass@256 is not an improvement you have yet to measure. It is a trade, and you have already made it.

Memory loops collapse too

None of this is exclusive to weight updates. Reflexion writes lessons in natural language to an episodic buffer after a failed attempt and reruns the task — 91% pass@1 on HumanEval against 80% for the GPT-4 baseline of the day, with no gradient step anywhere. Voyager grows a library of executable skills in Minecraft and retrieves them later, reaching 3.3× more unique items, 2.3× longer exploration distances and key tech-tree milestones up to 15.3× faster than the prior state of the art, again with frozen weights.

Both are self-improvement loops, and both have a concentration operator in them. A skill library that is only ever written from what already worked, and only ever read by nearest-neighbour retrieval, narrows the same way a policy does: the retrieved skill gets used, so it gets reinforced as the thing to retrieve, so the unusual skill three slots down is never tried. The failure is cheaper to fix — you can delete a memory, and you cannot un-train a weight — but it is the same failure, and it will not show up in pass@1 either.

Sampling two hundred answers and keeping the best one is search. It is simply the flattest search there is: two hundred branches from the root, no branch aware of any other, every branch run to full depth before anyone looks at it. Nothing learned in sample 14 reaches sample 15. Seeing it that way opens the rest of the design space, because the moment you let information move between branches you have a tree, and a tree spends the same token budget in a completely different shape.

The flat bank, and where it stops

Start with the flat version, because it is stronger than it sounds. Self-consistency (Wang et al., 2022) samples reasoning paths — 40 in the main results — and takes a majority vote over the final answers. On GSM8K with PaLM-540B that moves chain-of-thought prompting from 56.5% to 74.4%, a gain of 17.9 points, with no change to the model whatsoever.

Push the sample count and coverage keeps going. Large Language Monkeys (Brown et al., 2024) measured it directly: on SWE-bench Lite, DeepSeek-Coder-V2-Instruct resolves 15.9% of issues with a single attempt and 56% when allowed 250 samples, past the 43% that was the best single-attempt result at the time. Coverage grows roughly log-linearly in the number of samples across four orders of magnitude. The catch is in the same paper. Where an automatic verifier exists, that coverage is realizable; where one does not, majority voting and reward-model scoring plateau after a few hundred samples while coverage carries on climbing. The flat bank's ceiling is not generation. It is selection.

AlphaCode: filter, then cluster

AlphaCode (Li et al., Science, 2022) is the industrial answer to the selection problem, and its pipeline is worth walking through stage by stage, because each one is doing a specific job.

The competition allows a limited number of submissions, so the selector has to compress a million candidates down to ten without an oracle.

The filter is free signal: competitive programming problems ship with worked example cases in the statement, so every candidate can be executed against them before anyone spends a submission. Fewer than 1% of samples survive that — filtering removes about 99% of the bank. What remains is still thousands of programs and only ten may be submitted, so picking ten at random from the survivors mostly means submitting ten copies of the same popular mistake.

Clustering is the interesting part. AlphaCode trains a separate model to invent new test inputs from the problem statement — inputs with no known correct output, which is fine, because they are not being used to judge correctness. Every surviving program is executed on them, and programs producing identical outputs are grouped together. Two programs that are textually unrelated but semantically the same collapse into one cluster. Then it submits one program per cluster, largest cluster first. The budget now buys ten distinct behaviours instead of ten samples.

The scaling is the sobering part. On the CodeContests validation set the 41B model with clustering solves 21.0% at 10@1k, 26.2% at 10@10k, 31.8% at 10@100k and 34.2% at 10@1M. A thousandfold increase in samples bought 13 points. Log-linear is real, and log-linear is expensive. In ten simulated Codeforces contests with more than 5,000 participants each, that pipeline placed in the top 54.3% on average — an estimated rating of 1238, better than 72% of users who had entered at least one contest in the previous six months.

The block below runs that selection arithmetic on a bank scaled down from a million candidates to ten thousand, so it prints on one screen. The candidates are hand-written behaviour strings rather than compiled programs: what is real here is the filtering and clustering maths, not the code generation.

What the tree buys

Tree of Thoughts (Yao et al., 2023) makes the branches talk to each other. A problem is decomposed into intermediate states — partial solutions rather than whole ones. A proposer generates candidate next steps from a state, the language model is asked to evaluate the states it just produced, and a search keeps the promising ones and drops the rest.

On the Game of 24 with GPT-4 the spread is wide: input-output prompting 7.3%, chain of thought 4.0%, chain of thought with self-consistency at k = 100 9.0%, ToT with beam width 1 already 45%, and ToT at beam width 5 74%. Mini crosswords tells the same story at word level: 14% for IO, 15.6% for CoT, 60% for ToT.

Yao et al. (2023), GPT-4. Roughly the same budget, spent in a different topology. The tree is not buying more compute; it is buying a different arrangement of the compute it already had.

Three mechanisms explain the gap, and they are worth naming separately, because different tasks benefit from different ones:

1. It prunes. A flat sample commits its whole token budget before anything looks at it. A tree scores a partial state and can abandon it at depth two, so the budget concentrates on branches that are still alive.
2. It shares prefixes. Work done once at depth two is reused by every descendant. In a flat bank, a hundred samples that all open the same correct way each pay for that opening separately.
3. It backtracks. A left-to-right sample cannot un-say a bad third line. A tree can return to the second and take the other branch.

The costs are equally concrete. You need a state decomposition — and for many real tasks there is no honest answer to "what is a partial solution here?". You need a step proposer and a state evaluator, and the evaluator's error rate is now inside the loop at every node instead of once at the end, so a mediocre evaluator prunes the right branch early and confidently. And latency changes character: a bank of a thousand samples is embarrassingly parallel and costs one wall-clock unit, while a depth-four tree costs at least four, in series.

MCTS over agent states

The same idea extends past reasoning into acting. Language Agent Tree Search (Zhou et al., 2023) runs Monte Carlo tree search over agent states, with the language model playing three roles at once — the actor proposing actions, the value function scoring states, and the reflector writing up what went wrong — and environment feedback supplying the reward that keeps the search honest. It reports 92.7% pass@1 on HumanEval with GPT-4, and an average score of 75.9 on WebShop with GPT-3.5, which the paper describes as comparable there to gradient-based fine-tuning.

Read that last clause twice. It matched a fine-tuned system without a gradient step. Which is the whole point of this section, and also its limit.

Search improves the answer. It does not improve the model.

Every number above is bought again on the next request. Turn the search off and you are exactly where you started: the weights never moved, no library grew, nothing was retained. That is not a criticism — it is a different kind of purchase, and the economics of buying capability by the token at inference time are test-time compute's subject, not this page's.

Search becomes self-improvement at one specific moment: when its output is fed back into something that persists. There are exactly two places to put it, and they are the two axes this handbook opened with.

Into memory. The search result is written down and retrieved later. Voyager's skill library is this: the iterative search that finally produced a working program keeps the program, and the next task starts from a library rather than from nothing. AlphaCode's clusters are a weaker version — structure recovered from the search, then used once and discarded.

Into weights. The search becomes a data generator. rStar-Math (Guan et al., 2025) is the clearest worked example: extensive MCTS rollouts over 747k math problems produce step-by-step verified reasoning trajectories, those trajectories train the policy model, a process preference model trained on the same rollouts guides the next round's search, and the two are evolved together for four rounds. Qwen2.5-Math-7B goes from 58.8% to 90.0% on MATH; Phi3-mini-3.8B from 41.4% to 86.4%; on AIME 2024 the system solves 53.3% of problems, 8 out of 15. The search did the exploring. The weights kept the result, so the next round's search starts somewhere better.

Which closes the circle back onto the previous section. A tree search guided by a learned value function, trained on that same search's output, is the tightest version of the loop this handbook describes — generate, verify, filter, refit, with the generator itself getting sharper each round. Every warning in the loop that collapses applies to it with the exponent turned up: the value function is now both the filter and the thing the filter is training, so its blind spots do not merely survive between rounds, they steer where the next round is allowed to look.

A self-improvement loop rests on one claim: this week's agent is better than last week's. Everything above is machinery for making that claim true. This section is about making it checkable — because the benchmark most teams reach for cannot check it.

The problem is length. A typical benchmark item — one function, one question, one tool call — takes a competent person a minute or two. A production agent runs for hours and passes through hundreds of such moments. Two points on a short benchmark is perfectly compatible with your four-hour run failing exactly as often as it did last month, because what kills long runs is not what short items measure. Agent evals covers building the set and keeping it honest; this section is about what to read off it once you have one.

Measure the length of the job, not the score

METR's Measuring AI Ability to Complete Long Software Tasks (Kwa et al., March 2025) proposes a metric with a unit you can feel: time. The construction matters more than the headline, because you can rebuild it on your own tasks in an afternoon.

• Time the humans first. Every task carries a recorded human baseline — how long someone with the right skills actually took. METR's baseliners average around five years of relevant experience.
• Run the agent many times per task, scoring each attempt with the task's own mechanical grader.
• Fit one curve. A logistic regression predicting the agent's probability of success from the logarithm of the task's human duration.
• Read off the crossings. The duration where that curve passes 50% is the 50%-time horizon. Where it passes 80% is the 80% horizon.

The suite behind the original paper is 170 tasks — 97 from HCAST, 7 from RE-Bench, and 66 very short software actions — measured against more than 800 human baselines totalling 2,529 recorded hours of human work. Across frontier models from 2019 to early 2025, the fitted 50% horizon doubled roughly every 212 days, with a 95% bootstrapped confidence interval of 171 to 249 days — the figures as published in March 2025. The paper's current revision restates that fit as 207 days, 95% interval 166 to 240, so say which version you are quoting.

METR now maintains a live version on a larger suite, Time Horizon 1.1, and publishes the fitted numbers as data. There the all-time doubling sits at 188 days, and the fit restricted to models from 2023 onward is 129 days — a 95% interval of 104 to 158 days, or roughly four months per doubling over the recent stretch. Hold the trend loosely. Hold the per-model numbers tightly, because those are what you compare your own agent against.

The reason the unit matters is that anyone can check it against their own week. A benchmark score means nothing to the person deciding whether to leave an agent running overnight. The length of a job they recognise means a great deal.

Source: METR Time Horizon 1.1, the published dataset behind metr.org/time-horizons. Point estimates in minutes of human task time, rounded for display; the ratios are computed from the unrounded values, so recomputing one from the two rounded cells beside it can land a tenth off. The intervals are wide — Claude Opus 4.5's 50% horizon carries a 95% interval of 162 to 624 minutes — and METR notes that measurements above 16 hours are unreliable on the current suite, so the very top of the chart should be read as "long", not as a number.

The 80% column is the whole story

Read that table across, not down. Every model's horizon collapses when you ask it to be right four times in five instead of one time in two. Across the fourteen models in that dataset released in 2025 or later, the median ratio between the two columns is 5.1×, and the narrowest gap in the set — o3's — is 4.0×. The paper lands in the same range: it reports 80% horizons running 4–6× shorter than 50% ones. Its worked example, from the original 170-task fit rather than the larger suite behind the table above, is Claude 3.7 Sonnet — a 59-minute 50% horizon against an 80% horizon of about 15 minutes.

Say it in plain terms. Gemini 3.1 Pro's 50% horizon is a little over six hours of human work; its 80% horizon is an hour and a half. The model that can half-finish a working day can dependably finish a long meeting. The distance between those two sentences is your product.

A horizon is also not a promise about every task of that length. METR's own FAQ — written against an earlier fit, which is why it gives GPT-5 a horizon of 2 hours 17 minutes rather than the 203 minutes in the table above — takes the tasks a human expert needs 90 minutes to three hours for, and reports that a GPT-5 agent solves about a third of them every single time, fails about a third every single time, and is erratic on the rest. The headline number is a blend of three different populations — solved, unsolved, and flaky — and only the third one is cheap for your loop to move.

Then measure the work, not the task

Time horizons are measured on tasks built to be scorable. The other honest instrument drops that comfort and asks professionals to judge finished deliverables. OpenAI's GDPval (released September 2025, posted to arXiv in October) is built from the real output of working professionals — 1,320 tasks spanning 44 occupations across the nine largest sectors of US GDP, written by people with an average of 14 years in the job, with 220 of them open-sourced as a gold subset. The deliverables are real artifacts: documents, spreadsheets, decks, plans. A gold-subset task takes an expert 404 minutes on average — just under seven hours.

Two instruments, then, failing in opposite directions. One asks whether the agent can keep going long enough; the other asks whether what came out was any good. You want both, and a benchmark score on its own is neither.

The grading is the part to copy. An expert in that occupation sees the request, the reference files, and two unlabelled deliverables, and ranks them without being told which came from a model. The paper collects three model samples per prompt and puts three graders on each, which is nine blinded comparisons per prompt per model across all 220 tasks. A single comparison takes a grader over an hour.

The paper's headline: on the gold subset, 47.6% of Claude Opus 4.1's deliverables were graded better than (a win) or as good as (a tie) the human expert's — the best of the models it tested. Not "as good as a professional". It means that in blinded head-to-head against the specific professional who did that job, a grader called it for the model or called it even in a little under half the pairings.

Source: the GDPval paper's speed-and-cost table (Table 2), whose win-rate column covers only the OpenAI models it priced — hence no Claude, Gemini or Grok row. GDPval counts a deliverable graded as good as the expert's alongside one graded better, which is the same measure as the 47.6% above, so the column and that figure are on one scale; Claude Opus 4.1's number is simply the one the paper states in prose rather than plotting. Read the ladder for its slope, not its absolute level; the leaderboard has moved since.

Why long runs fall apart: multiplication

Both instruments are measuring the same underlying thing, and it is not intelligence. It is compounding.

One number raised to a power. That is the entire reason a capable model fails a long job.

Put real numbers in it. At 99% per step — a rate most people would call excellent — a 50-step run finishes 60.5% of the time and a 100-step run finishes 36.6%. At 95% per step, the 50-step run finishes 7.7% of the time. Run it backwards and it gets worse: to land 90% of 50-step runs you need 99.79% per step, which is one slip per 475 steps. That is not a number you reach by tuning a prompt.

Independence is the assumption carrying that formula, and it is wrong in both directions. Real agents retry, so some failures cost tokens instead of the run. Real failures are also correlated — one bad plan poisons every step after it — so one mistake can be worth many. Treat pⁿ as the shape of the problem, never as a forecast of your system. The block below is that arithmetic and nothing else: no model, no sampling, no training run.

Reliability is the binding constraint, not capability

Look at what the arithmetic implies about where to point your loop. Going from 90% to 95% per step lifts a 50-step run from 0.5% to 7.7%. Going from 95% to 99% lifts it from 7.7% to 60.5%. Going from 99% to 99.9% lifts it to 95.1%. Every nine you add is worth more than the one before it, and none of those moves teaches the model anything it could not already do. The model can do the step. It does it 95 times in 100. Your loop's entire job is the other 5.

That is why a self-improvement loop aimed at the lowest benchmark score usually disappoints, and one aimed at the flakiest step usually does not. Find the step that fails most often in your traces, and spend memory, context and weights on that one step until it stops. Then find the next one. Long-horizon agents is the companion piece on what else breaks when a run lasts hours.

Build your own two instruments

You cannot run METR's suite or GDPval. You can run both methods, cheaply, on your own work:

• Your horizon. Take 50 or more real tasks off your actual queue. Record how long a competent person takes on each. Run the agent five to ten times per task. Fit success against the logarithm of that duration and report two numbers — the 50% and the 80% crossing. Track the 80% one. It is the number your users experience.
• Your win rate. Freeze 50 to 200 real tasks. For each, keep one deliverable made by your best human. Have someone who does the job rank the two, blind, against a rubric written before the model ran. Your improvement claim becomes a win-or-tie rate on the frozen set that moved — a sentence your loop cannot manufacture by learning to please your verifier.

One caveat both sources state about themselves, which applies double to you. A METR task is written to stand on its own, so that a baseliner and an agent start from the same place and neither is punished for not knowing the house rules. Your queue is the opposite: half of what makes a ticket easy is knowing which service owns it and who broke it last time. METR's own advice is to read a two-hour task as what someone with "low or no prior context (like a new hire or freelance contractor) could complete in 2 hours". Your production tasks come loaded with context, so your horizon will not match anyone's published figure — and it does not need to. The number exists to compare your agent to your agent, last month.

Here is the order to do it in. It is deliberately front-loaded with measurement, because every failure mode above — reward hacking, drift, a filter that lets its own mistakes through — is invisible without it, and every one of them is cheap to prevent and expensive to unwind. Each step names the action and points back at the section that argues for it.

1. Name one task family and one deliverable.

   Not "the agent". One repeating job — triage this ticket type, draft this document, fix this class of bug — and one written sentence describing what a finished one looks like, specific enough that a stranger could apply it. If you cannot write that sentence, nothing downstream has a target.

2. Measure a horizon before you measure a score.

   Pull 50 or more real tasks off the queue, record how long a competent person takes on each, run the agent five to ten times per task, and fit success against the logarithm of that duration. Report the 50% and the 80% crossing. This pair is your baseline and your only defence against shipping a change that helped the benchmark and hurt the product.

   Why: 12 · Knowing it worked

3. Build the verifier before you build the loop.

   No trace may enter any update until something mechanical can say correct or not: tests that run, a schema that validates, a diff against a known answer, a rubric grader. If you cannot verify the task, you cannot safely improve on it — stop here and change the task.

   Why: 05 · Verification, 04 · Selection is the bottleneck

4. Measure the verifier itself.

   Hand-label 100 runs and score your verifier against your labels. Its false-positive rate — how often it blesses a wrong run — is the hard ceiling on everything downstream, because those are exactly the traces that get baked in. A verifier you have not measured is a number you have not earned.

   Why: 05 · Verification

5. Log complete, replayable traces from day one.

   Every prompt, tool call, tool result, final output, verifier verdict, and the version of the model, the prompt and the tools that produced it. A trace you cannot replay is an anecdote. This is also the only artifact that makes a post-mortem possible six weeks later.

   Why: 07 · What you keep

6. Find where the runs actually die.

   Bucket failures by step index and by cause, and sort. The distribution is almost never flat — one or two steps carry most of the loss. Fix the tallest bar. Improvement spent anywhere else is invisible in the compounding arithmetic.

   Why: 12 · Knowing it worked

7. Reach for the cheapest reversible fix first.

   In order: prompt and tool descriptions, then scaffold and retry logic, then memory, then weights. Each rung costs more and reverses harder than the one below it. Most teams skip to the top rung and pay for a training run to fix a tool description.

   Why: 08 · Where the improvement lands

8. Land recurring specifics in memory.

   This customer's preference, this repo's build command, this API's quirk, this failure and its remedy. Anything narrow, anything you might have to revoke, anything you are not yet sure about. Write it, retrieve it, and keep the delete button one click away.

   Why: 08 · Where the improvement lands

9. Land broad, verified behaviour in weights — and only then.

   The signal that weights are the right home is memory repeating itself: the same kind of note written a hundred times for a hundred different tasks. Train on filtered-correct traces only, at a bar strictly higher than memory's, because this is the rung you cannot climb back down.

   Why: 08 · Where the improvement lands, 09 · Closing the loop

10. Freeze an eval set the loop can never touch.

    Held out of training, held out of memory, held out of prompt examples, and owned by someone whose job is not shipping the loop. The day it leaks, your only independent measurement quietly becomes a training metric and you will not be told.

    Why: 12 · Knowing it worked

11. Canary with enough runs to mean something.

    Route a slice of live traffic to the candidate, keep both versions loadable, and size the sample to the regression you care about rather than to your patience. Ten runs is not a canary: a model that has silently dropped from 90% to 80% still clears a 90% bar 37.6% of the time. Catching a five-point drop nineteen times out of twenty takes about 118 runs. The sizing block below does that arithmetic for your own numbers.

    Why: 12 · Knowing it worked

12. Monitor four numbers, not one.

    Verifier pass rate on live traffic. The 80% horizon on the frozen set. Memory store size against memory hit rate. And a weekly human-graded win-or-tie sample. The fourth exists precisely because the first three can all improve while the product gets worse.

    Why: 12 · Knowing it worked

Every page below owns a piece this one deliberately did not re-teach. They are grouped by the question they answer, so you can go to the one you are actually stuck on.

Questions people actually ask