DeepSeek-R1,
explained.

Rung 4 establishes that reasoning can be trained and does not say how. This rung's change is to show that the recipe can be far simpler than anyone expected and to publish it. The starkest version is the variant trained with no supervised fine-tuning at all: reinforcement learning applied directly to a base model, with a reward that checks only whether the final answer is correct and whether the output is in the required format. No human reasoning demonstrations, no step-level labels, no learned reward model to game.

What emerges under that reward is the finding. The model's responses get longer on their own over training — nothing rewards length — and behaviours the ladder below had to scaffold appear inside a single generated stream: reconsidering an approach, checking its own work, backtracking from a line that is not working. Rung 3 built that with an external search; here it is a policy the model arrived at because it raised the probability of being right.

Stated in the units this ladder compares on: rung 9 of the post-training ladder argues that supervising the process beats supervising the outcome, and buys that with an enormous annotation effort. This rung reaches comparable reasoning behaviour with outcome supervision only, paying in reinforcement-learning compute instead of in labels. The released version adds a small amount of cold-start data before the reinforcement learning, for reasons the standing block explains, and the weights and method are open.

A standard chat model answers fast — it predicts a response in one flowing pass. That's fine for "summarize this email," but it stumbles on problems that need genuine multi-step work: hard math, tricky code, layered logic. Answering such a question well requires thinking — trying an approach, checking it, backing up, trying another — before committing to a final answer.

OpenAI's o1 had shown that letting a model generate a long internal chain of thought before answering dramatically boosts reasoning — but the recipe was closed. The open question DeepSeek-R1 set out to answer: how do you actually train a model to think like that? And could you do it without a mountain of hand-written reasoning examples?

The boldest experiment came first: DeepSeek-R1-Zero. Take a base model and skip the usual supervised fine-tuning on example reasoning entirely. Instead, apply reinforcement learning directly, with a shockingly simple reward — mostly just: is the final answer correct? (checkable for math and code), plus a small reward for putting the reasoning in the right format.

Crucially, nothing rewards the reasoning itself — no human ranks the steps, no example solutions are copied. The model is only ever told whether it landed on the right answer. Everything about how to get there is left for it to discover. It's the leanest possible teaching signal: outcome only.

The result was the paper's headline. Left to maximize correctness, the model discovered — with no instruction to do so — that thinking longer helps. Over training, its responses grew from short guesses into extended chains of reasoning: laying out steps, computing intermediate results, and exploring more than one path before answering. Accuracy on hard math and coding benchmarks climbed steeply alongside this growing "thinking time."

This is a striking demonstration that a complex skill can be incentivized rather than imitated. Nobody wrote reasoning examples for R1-Zero to copy; the reasoning behavior was an emergent strategy for earning more reward. Give a capable base model the right pressure, and the sophisticated behavior falls out.

You can see the emergence fall out of the numbers. If expected accuracy rises with thinking length but each step costs a little compute, reward pressure drives the chain to exactly the length where the next step stops paying for itself:

No one wrote a single reasoning example; long chains appeared purely because they earn more reward. The runnable version below grows the chain and shows the reward is ungameable.

The most talked-about observation: during training, the model spontaneously learned to stop and reconsider. Mid-solution, it would effectively say "wait — let me rethink this," abandon a flawed approach, and try again. It also learned to spend more thinking on harder problems and less on easy ones, budgeting its own effort.

None of this self-correction was designed in. It emerged because catching your own mistakes gets more answers right, and the reward found it. The authors called it an "aha moment" — a genuinely surprising sign that reflective, self-checking behavior can arise from pure outcome-based reinforcement.

R1-Zero reasoned impressively but communicated poorly. With only correctness rewarded, its chains of thought were often hard to read and sometimes mixed languages mid-answer — nobody had rewarded clean, human-friendly output. Powerful, but not pleasant to use.

So the full DeepSeek-R1 added structure around the pure-RL core. First, fine-tune the base model on a small batch of high-quality cold-start reasoning examples to instill a clean style. Then run the large-scale RL for reasoning. Then a round of rejection sampling and more training to polish helpfulness and readability. The emergent reasoning is preserved; the output becomes coherent and well-behaved.

Running large-scale RL is expensive, so DeepSeek showed a cheaper path to spread the capability: distillation. Use the strong R1 to generate many high-quality reasoning traces, then simply fine-tune smaller open models (in the 1.5B–70B range) on those traces. No RL needed for the small models — they learn to reason by imitating R1's worked solutions.

The distilled models inherited a large share of R1's reasoning — small models that punch far above their size on math and code. It's strong evidence that reasoning skill, once created, transfers cheaply, putting capable reasoners within reach on modest hardware.

The clean RL recipe works best where answers are automatically checkable — math, code, formal logic. For open-ended tasks with no crisp "correct" signal (writing, judgment, taste), outcome-based RL is far harder to apply, so R1's edge is narrower there. Long chains of thought also cost real test-time compute: thinking longer means more tokens, more latency, more money per answer.

And reasoning fluency isn't truthfulness — a confident, well-structured chain of thought can still reach a wrong conclusion, and the visible "thinking" doesn't guarantee the model actually followed that logic. Emergent self-correction reduces errors; it doesn't eliminate them.

DeepSeek-R1 opened the reasoning era to everyone. It reproduced o1-class reasoning with an openly published recipe and open weights, and — through R1-Zero — showed something conceptually deep: advanced reasoning can be incentivized into existence by rewarding correct outcomes, not painstakingly demonstrated. That reframes how we think about teaching models hard skills.

It also cemented test-time compute — letting a model think longer at inference — as a first-class lever alongside making models bigger. Together with the cheap distillation of reasoning into small models, R1 pushed the whole field toward reasoning-first systems and made them broadly accessible, not just a frontier-lab privilege.

DeepSeek-R1 is a report from DeepSeek-AI, released in January 2025 and later published in Nature. It describes two models trained from the same base, DeepSeek-V3-Base, a mixture-of-experts model with 671B total and 37B active parameters, plus a cheap way to pass the result on to smaller models.

R1-Zero: reinforcement learning straight on the base model

1. Fix a template, not a style. Every prompt is wrapped in one short instruction: the assistant thinks first and then answers, with the reasoning between <think> and </think> tags and the answer between <answer> tags. Nothing in it asks for reflection or any particular strategy, so whatever reasoning style appears is learned.
2. Reward with rules, not a model. Two rewards. Accuracy: a math answer must appear in a required format, such as inside a box, so it can be checked against the reference, and code is compiled and run against test cases. Format: the reasoning must sit inside the think tags. The authors deliberately avoided neural reward models, outcome or process, because a large policy learns to hack them and retraining them adds cost and complexity.
3. Optimize with GRPO. For each question, sample a group of answers from the current policy, score each with the rules, and use each answer's reward relative to its own group, standardized by the group's mean and standard deviation, as its advantage. There is no learned critic, which removes a second model as large as the policy from the training loop.
4. Repeat for thousands of steps. Average response length grows steadily without being rewarded, from hundreds of tokens to thousands, and AIME 2024 accuracy climbs with it.

R1: four stages

1. Cold start. Fine-tune V3-Base on thousands of long chain-of-thought examples, gathered by few-shot prompting with long examples, by prompting for detailed answers with reflection and verification, and by having annotators clean up R1-Zero outputs. Each example ends with a readable summary.
2. Reasoning RL. The R1-Zero procedure on math, code, science and logic, with one added reward for language consistency: the share of the chain written in the target language. In the authors' ablation it costs a little accuracy and makes the output readable.
3. Rejection sampling and fine-tuning. From the converged checkpoint, sample several responses per prompt and keep only correct ones, judging some with DeepSeek-V3 against a reference answer, and drop chains with mixed languages, long paragraphs or code blocks: about 600k reasoning samples. Add about 200k non-reasoning samples, such as writing, factual QA and translation, from the DeepSeek-V3 pipeline. Then fine-tune V3-Base again, not the RL checkpoint, for two epochs on the roughly 800k samples.
4. RL for all scenarios. A second RL stage mixes rule rewards for reasoning prompts with preference reward models for general ones. Helpfulness is judged on the final summary only; harmlessness on the whole response, reasoning included.

Distillation and inference

The same 800k samples fine-tune open dense models, Qwen2.5 at 1.5B, 7B, 14B and 32B and Llama at 8B and 70B, with supervised learning only and no RL stage. At evaluation, every model generates up to 32,768 tokens at temperature 0.6 and top-p 0.95, and pass@1 is averaged over several samples per question.

For each question q, GRPO samples a group of G outputs o₁ … o_G from the old policy and maximizes:

• ri — the rule-based reward for output i: accuracy plus format
• Ai — the advantage, how much better output i scored than its own group; it replaces PPO's learned value function
• ρi, ε — the probability ratio and PPO's clip, which caps how far one update can move the policy
• β, πref — the KL penalty weight and the frozen reference model it anchors to

If every output in a group earns the same reward, every advantage is zero and the question contributes no reward signal, so only questions the model sometimes solves teach it anything. The KL estimate is never negative and is zero when the two policies agree. Liu and colleagues later argued, in their 2025 Dr. GRPO analysis, that dividing by the group's standard deviation, and by response length in the token-level form of GRPO, biases training, for example letting long wrong answers grow; they proposed dropping both.

Evaluation used reasoning benchmarks (AIME 2024, MATH-500, GPQA Diamond, LiveCodeBench, Codeforces) and general ones such as MMLU, against OpenAI's o1 releases, GPT-4o, Claude 3.5 Sonnet and DeepSeek-V3. Pass@1 is averaged over several samples per question at temperature 0.6.

R1 matched o1-1217 on math and came close on competitive programming, and trailed it on GPQA Diamond, 71.5% against 75.7%. R1-Zero's 86.7% with majority voting was above the earlier o1-0912 release under the same kind of voting.

Negative results. The key ablation is the last row: the R1-Zero recipe run directly on Qwen2.5-32B-Base for more than 10,000 steps reached roughly QwQ-32B-Preview's level, while distilling R1 into the same base scored far higher. The authors also report two methods that did not work. A process reward model was hard to define for general reasoning, hard to label, and open to reward hacking once learned. Monte Carlo tree search over reasoning steps faced a search space far larger than a board game's and a value model that was hard to train, so it did not keep improving the model. R1 itself was weaker than DeepSeek-V3 at function calling, multi-turn dialogue, role-play and JSON output, sensitive to prompts, with few-shot prompting hurting it, and barely better on software-engineering benchmarks, where slow evaluation limited the RL.

The admissions are unusually concrete. The pure-reinforcement-learning variant produces reasoning that is hard to read and mixes languages mid-chain — which is exactly what a reward that only checks the final answer should be expected to produce, since nothing in it ever asks the reasoning to be legible to a human. The cold-start data exists to fix that, and the paper says so. It also reports being weaker on some non-reasoning capabilities, so this is a trade rather than a free gain.

The most interesting reported negative concerns small models: distilling from the large reasoning model beat running the same reinforcement learning directly on a small one. The reasoning behaviour did not bootstrap itself at small scale — it had to be produced by something large and then transferred. That is a finding about where this capability comes from, and it echoes rung 1's threshold from the other end of the ladder.

How it held up: it was reproduced widely and quickly, which is the strongest possible contrast with rung 4's position, and it turned reasoning training from a frontier-lab capability into something many groups could run. The tension it leaves open is genuine and unresolved: outcome-only reward at sufficient scale produced what process supervision was designed to produce, and whether that means process supervision was unnecessary or that expensive reinforcement learning is rediscovering it the slow way is not settled by either result.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

It began as a prompting trick and ended as a training objective. Read in order, the progression from "let's think step by step" to a model trained to.

1. 1Chain-of-Thought Prompting2022
2. 2Self-Consistency2022
3. 3Tree of Thoughts2023
4. 4OpenAI o12024
5. 5DeepSeek-R12025

All 110 papers, in 19 ladders →