STaR,
explained.

Rung 3 generates tasks and filters them for diversity. This rung generates reasoning and filters it for correctness, which is a much stronger filter because it is not a heuristic — it is the answer key. The loop: prompt the model with a few worked examples, let it produce a chain of reasoning and a final answer for each training problem, keep only the chains whose final answer is right, fine-tune on those, and repeat with the improved model. Each round the model solves problems the previous round could not, which produces training data the previous round could not.

That loop stalls in an obvious place: problems the model never gets right generate no training data at all, so the hardest problems — the ones worth learning from — are exactly the ones that contribute nothing. The fix is the paper's cleverest move and it runs the inference backwards. Give the model the correct answer as a hint and ask it to produce the reasoning that leads there. A chain generated with the answer in hand is then used as training data without the hint, so at training time the model sees a hard problem paired with a plausible route to its solution.

Stated in the units this ladder cares about: the supervision signal costs nothing per example beyond a known answer — no human writing reasoning, no preference labels, no reward model. Where rung 1 needed contractors and rung 9 needs an expensive annotation effort, this needs a dataset of problems with answers, which already exists for every subject that has ever been examined.

Chain-of-thought showed that models reason better when they write out intermediate steps. The natural follow-up is to train a model to always do that well — but that seems to need a big dataset of high-quality reasoning traces, and those are scarce and costly to write. What you usually have is only the easy part: questions paired with their final answers, no worked solutions.

STaR's move is to close that gap by having the model generate its own rationales and using the known answers as a filter. When the model reasons its way to the correct answer, that reasoning is probably decent — good enough to learn from. When it reasons to a wrong answer, discard it. You never wrote a single rationale; you just judged the model's own reasoning by whether it worked. That turns a pile of question-answer pairs into reasoning training data, for free.

The core loop is three steps, run over your dataset of problems:

Step 2 is the whole trick: the ground-truth answer is a free, automatic quality check that selects which self-generated reasoning to trust. The model isn't told how to reason — only whether its reasoning arrived somewhere correct, and it learns from the ones that did. After fine-tuning on the keepers, the model is a bit better, so the next round it generates more correct rationales, which makes it better still. The runnable version below shows the filter keeping only correct-answer rationales and accuracy climbing each round.

There's a flaw in the plain loop: hard problems never contribute. If the model can't solve a problem, it never produces a correct rationale for it, so that problem is silently excluded from training — forever. The model only ever learns from problems it could already almost do, and the hard ones stay hard. That caps how far bootstrapping can go.

STaR's fix is rationalization. For a problem the model got wrong, give it the correct answer as a hint and ask it to produce a rationale that reaches that answer — reasoning backward from the solution. If it can construct such a rationale, that reasoning trace is added to the training set (with the hint removed). Now the hard problems can contribute reasoning examples too, dramatically widening what the model learns from.

The runnable below makes this concrete: direct solving covers a fraction of problems, and rationalization pulls in a big chunk of the ones it missed — so the training set spans the hard problems the plain loop would abandon.

Run the loop repeatedly and it bootstraps: each iteration's improved model generates better, more numerous rationales, which fine-tune an even better model. It's a self-improvement flywheel powered by nothing but question-answer pairs and the model's own generations. The improvement isn't unbounded — it tapers as the model approaches the limit of what this data and process can teach — but the early rounds compound quickly.

What makes it work is the asymmetry between generating a rationale and checking it: the model can't always reason correctly, but you can always check whether its answer is right. That gap — hard to produce, easy to verify — is exactly what STaR exploits, turning a cheap verifier (does the answer match?) into a training signal for the expensive skill (reasoning). It's the same asymmetry that underlies a great deal of later reasoning work.

STaR needs a pretrained model, a dataset of questions with known answers and a handful of worked examples. The paper used GPT-J, a 6B-parameter model, on three datasets: generated n-digit addition problems, CommonsenseQA (five-way multiple choice, 9,741 training and 1,221 dev questions) and GSM8K (7,473 training and 1,319 test problems).

1. Prompt with a few rationales. A fixed few-shot prompt shows the format: question, rationale, answer. For CommonsenseQA it was 10 questions based on the chain-of-thought paper's exemplars, with rewritten rationales; for arithmetic, 10 scratchpad examples per digit length.
2. Generate. The current model writes a rationale and an answer for every training question, using greedy decoding.
3. Filter by the answer key. A rationale is kept only if its final answer matches the known answer. No other check is applied.
4. Rationalize the misses. For questions answered wrongly, the correct answer is shown as a hint and the model writes a rationale that reaches it. In CommonsenseQA the correct option is marked as the right one in the list of choices; in arithmetic the target answer is appended. A rationalized example is kept only if it lands on the correct answer, and it is stored without the hint, so the training example reads as if the model had reasoned there unaided.
5. Fine-tune from the original model. Each outer iteration fine-tunes the original pretrained GPT-J on all kept examples, rather than continuing from the previous iteration's checkpoint, to avoid overfitting. For CommonsenseQA, training started at 40 steps and grew by 20% each iteration.
6. Repeat. The new model generates the next round's rationales. A better model keeps correct rationales for more questions, and the loop runs until performance stops improving.

Why the decoding is greedy

The authors tried sampling rationales at higher temperatures, 0.5 and 0.7, to get more varied training data. It backfired: higher temperature raised the chance of reaching the correct answer through flawed reasoning, which the filter then kept, and models trained on that data did worse; on arithmetic the scratchpads degenerated. On CommonsenseQA the risk is built in, since a random guess among five options is right 20% of the time and simple similarity heuristics reach about 30% with no reasoning at all.

STaR as approximate reinforcement learning

The paper frames the loop as an approximation to a policy-gradient method whose reward is 1 when the final answer is correct and 0 otherwise. Filtering discards exactly the zero-reward samples, and fine-tuning on the rest follows that gradient. The approximations are greedy decoding instead of sampling, many fine-tuning steps on each round's fixed data, and rationalization, whose samples come from a hint-conditioned distribution the model never sees at test time. That last piece is not part of the on-policy gradient (the paper suggests framing it as an off-policy estimate, with the hint-augmented model as the proposal distribution), and it is what lets failed questions contribute at all.

• xi, yi — a training question and its known answer.
• r̂i, ŷi — the rationale and answer the model M generates.
• 1(ŷi = yi) — the reward: 1 if the answer matches, 0 otherwise. Zero-reward samples contribute nothing to the gradient, which is why discarding them loses nothing.
• ∇ log pM(ŷi, r̂i | xi) — the fine-tuning gradient on the kept text, rationale and answer together.

Two consequences follow. A question the model never answers correctly has near-zero expected reward and contributes no gradient at all, which is the gap rationalization fills with samples from a hint-conditioned model rather than from p_M. And the reward checks only ŷ, so a wrong rationale that happens to land on the right answer is reinforced exactly as strongly as a sound one.

The comparisons isolate two questions: do rationales help at all, measured by fine-tuning GPT-J to output answers directly, and does rationalization help, measured by running STaR without it. A roughly 30 times larger GPT-3 fine-tuned to answer directly is the reference point.

The training-data column is the share of training questions that ended up with a kept rationale; rationalization raised it from 69.7% to 86.7%, alongside its 3.7-point accuracy gain. On GSM8K, STaR reached 10.7% against 5.8% for direct-answer fine-tuning and about 3% for few-shot prompting, but it kept rationales for only 28.7% of training problems, a reminder that the loop can only amplify what the model can sometimes already do. On arithmetic, STaR with rationalization reached 89.5% overall after 16 iterations, against 76.3% for a baseline trained on 10,000 examples without rationales; without rationalization, accuracy rose in stages, one digit length at a time.

A small human study added a qualitative check: crowdworkers ranking rationales for 50 questions that both methods answered correctly preferred STaR's rationales to the few-shot prompt's more often than not. The authors do not claim this shows human-level rationale quality.

The admission is precisely located and it is the reason rung 9 exists. Filtering on the final answer accepts any chain that arrives at it, including chains that are wrong, incoherent, or that reach the right answer for the wrong reason — and the backwards-reasoning trick makes this worse by construction, because a chain written with the answer already known is under no pressure to be the reasoning that would have found it. Post-hoc justification is a category of output humans produce readily, and nothing in this filter excludes it. A lucky guess with a confident explanation is indistinguishable from understanding, to this filter, and gets reinforced.

It also needs known answers, so it is confined to verifiable domains, and each round's fine-tune is trained on the previous round's distribution, which is a self-reinforcement loop with no external correction.

How it held up: as the template for everything that followed. Generate attempts, keep what verifies, train on the survivors, repeat is the shape of the reasoning-model wave — and the largest change since was to swap the supervised fine-tune for reinforcement learning on the same verified signal, which makes the loop continuous rather than round-based. Read this rung as the origin of the idea that a model can be its own source of training data for reasoning, and rung 9 as the objection that the filter is measuring the wrong thing.

STaR reads, in hindsight, like a blueprint for the reasoning-model era. Its loop — generate candidate reasoning, keep what's verified correct, train on it, repeat — is precisely the shape of the reinforcement-learning-from-verified-answers methods (like GRPO with verified-answer rewards, the setup DeepSeek-R1 used) that trained the current generation of reasoning systems. STaR did it with supervised fine-tuning on filtered samples; later work does it with RL and reward signals, but the core self-improvement idea is the same.

Its enduring lesson is the generate-verify asymmetry as an engine for self-improvement: whenever producing a solution is hard but checking it is easy, a model can bootstrap by generating many attempts and learning from the ones that pass the check. Reasoning with checkable answers is the clearest case, which is why STaR started there — and why that pattern now underlies how frontier models learn to think. It also sits naturally beside process reward models, which sharpen the "verify" side that STaR relies on.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →