GRPO,
explained.

Rung 6 removes reinforcement learning because it is expensive and unstable. This rung keeps it and removes the part that made it expensive. Policy gradients need a baseline — an estimate of how good a state already is — so that an action can be credited for being better than expected rather than merely for preceding a positive outcome. PPO supplies it with a learned value network, which for a language model means a second network of comparable size to hold in memory and train.

The change is to get the baseline from the samples instead of from a model. Sample a group of answers to the same prompt, score them all, and use the group's own mean as the baseline — each answer's advantage is how much better it did than its siblings. This is available because of a property of the tasks involved: all the method needs is a scalar score per sample. DeepSeekMath itself scored samples with a learned reward model; for mathematics and code a score can instead come from checking the answer (the route DeepSeek-R1 later took), which makes generating twenty attempts and grading all of them cheap and harder to game.

Stated in units: one model of policy size leaves memory and one training process disappears, and the compute that was spent maintaining a critic moves into sampling more attempts per prompt. That is a favourable exchange for a specific and important reason — sampling is embarrassingly parallel and needs no state, while training a critic alongside a policy is a coupled optimisation that can fail in ways that are hard to diagnose.

Reinforcement learning is how you teach a model from outcomes instead of examples: generate an answer, score it, nudge the weights toward what scored well. The catch is the nudging. A policy gradient needs to know not just "was this good?" but "was this better than expected?" — otherwise every sample looks equally worth reinforcing and training is hopelessly noisy.

"Better than expected" requires a baseline: an estimate of the expected reward. The standard answer — PPO, inherited from InstructGPT and the RLHF lineage — learns that baseline with a value network (the critic). For LLMs, the critic is typically another copy of the model. Training one 7B policy means holding a 7B critic beside it, training both, syncing both. The bill doubles before you generate a single token.

Four networks sit in memory during classic RLHF-style PPO: the policy being trained, the frozen reference model (to keep outputs from drifting too far), the reward model (what scores answers), and the critic (what predicts expected score, token by token). The critic exists purely to compute the advantage — how much better this output was than the expectation.

The critic is also the fragile one: it must learn to predict rewards for partial generations — a moving target that shifts every time the policy improves. A mis-trained critic silently poisons the advantages, and with them, the whole run.

Group Relative Policy Optimization deletes the critic with one observation: if you sample several answers to the same prompt, you already have an estimate of the expected reward — the group's average. No learned predictor needed; the cohort grades itself.

Each answer's advantage is simply its reward minus the group mean (normalized by the group's spread). Beat your siblings → reinforced. Trail them → suppressed. A KL penalty against the reference model still keeps the policy from wandering off. The result is PPO's learning signal with roughly half the memory footprint and none of the critic's training instability.

The whole advantage computation is one line — no network, just the group's own statistics:

Two properties fall straight out of that formula. Subtracting the mean makes the advantages sum to zero — every step reinforces the above-average answers exactly as much as it suppresses the below-average ones, a built-in baseline. And dividing by the group's spread means an answer is judged only against its own siblings, so problem difficulty cancels: a mediocre answer to a brutal problem can still be reinforced if it beat the pack. The runnable version below shows the same 0.5 reward earning a positive advantage on a hard prompt and a negative one on an easy prompt.

GRPO's sweet spot is domains with verifiable rewards: math answers check, code runs or doesn't, proofs verify. The signal is crisp and ungameable, so the group comparison measures something real. This pairing — group-relative advantages plus checkable correctness — is what later let reasoning be incentivized into existence rather than imitated.

The within-group comparison is also exactly the right granularity. An absolute score can't distinguish "hard problem, decent attempt" from "easy problem, sloppy attempt." Comparing attempts at the same prompt cancels problem difficulty out — the model learns which of its own strategies works, prompt by prompt. It's a tournament among siblings, not a grade against an absolute curve.

The paper's vehicle was DeepSeekMath 7B. Step one had nothing to do with RL: continue pretraining a code model for 500B tokens, 56% of them from a 120B-token corpus of carefully filtered math scraped from the web — a data-curation feat in itself. Step two: instruction-tune. Step three: GRPO.

A 7B open model approaching the math performance of vastly larger closed models was the headline. The quieter, more consequential result was the recipe: quality data plus cheap, stable RL substitutes for scale.

A year later, DeepSeek pointed GRPO at a stronger base model, used rule-based accuracy and format rewards, and skipped supervised reasoning data entirely. The result was DeepSeek-R1-Zero — chain-of-thought reasoning, self-checking, the famous "aha moment," all emerging from GRPO pressure toward correct answers. GRPO wasn't a footnote in that story; it was the engine that made RL cheap and stable enough to run at that scale.

The broader field followed. Group-relative, critic-free RL became the default recipe for reasoning training across open labs, spawning a family of refinements, and "RL with verifiable rewards" became the standard phrase for the whole paradigm — the training-side twin of test-time compute.

GRPO inherits RL's oldest failure mode: reward hacking. If the reward is even slightly gameable — a weak checker, a length bias, a format loophole — group pressure will find and exploit it ruthlessly. It also leans on cheap sampling: every training prompt costs G full generations, trading the critic's memory for extra inference compute.

And it shares outcome-based RL's boundary: where correctness isn't checkable — taste, judgment, open-ended writing — the crisp signal disappears, and you're back to learned reward models with all their softness. GRPO didn't solve alignment for fuzzy domains; it made RL practical where the answer key exists.

GRPO was the last of three stages that produced DeepSeekMath 7B. DeepSeek-Coder-Base-v1.5 7B was first continually pretrained for 500B tokens, 56% of them from the 120B-token DeepSeekMath Corpus filtered out of Common Crawl, then instruction-tuned on 776K math examples in chain-of-thought, program-of-thought and tool-integrated formats. GRPO then ran on about 144K chain-of-thought questions drawn from that instruction data, all related to GSM8K and MATH.

1. Train a reward model. The reward came from a learned model, not from checking answers. It was initialized from DeepSeekMath-Base 7B and trained at a learning rate of 2e-5 on data built following Math-Shepherd, a method that labels reasoning steps automatically. The experiments use both outcome rewards for whole solutions and process rewards for individual steps.
2. Sample a group. For each question q, the current policy samples G = 64 solutions of up to 1,024 tokens.
3. Score every sample. Under outcome supervision each solution gets one reward from the reward model; under process supervision each reasoning step gets its own.
4. Normalize inside the group. With outcome supervision, every token of solution i gets the same advantage: its reward minus the group mean, divided by the group standard deviation. With process supervision, all step rewards in the group are normalized together, and a token's advantage is the sum of the normalized rewards of its own step and every later step.
5. Take a clipped policy-gradient step. Each token's update uses the PPO probability ratio, clipped to [1 − ε, 1 + ε]; per-token terms are averaged over each solution's length and then over the group. The training batch size was 1,024, the policy learning rate 1e-6, and each exploration stage was followed by a single policy update.
6. Regularize with KL in the loss. PPO-style RLHF subtracts a KL penalty inside the reward. GRPO instead adds β·D_KL(π_θ ‖ π_ref) directly to the loss, with β = 0.04, using a per-token estimator that is unbiased and never negative. Keeping the KL out of the reward keeps it out of the advantage calculation too.
7. Iterate, optionally. In iterative GRPO, the policy's samples become new training data for the reward model, which keeps 10% of its earlier data in a replay buffer, and the reference model is reset to the current policy before the next round.

Where the reward came from

DeepSeekMath's GRPO ran against a learned reward model. Rule-based rewards that check the final answer and the output format belong to DeepSeek-R1, published about a year later, which paired GRPO with such verifiable rewards. The group baseline needs only one scalar per sample, so it works with either kind; what changes is how far the reward can be trusted and how easily it can be gamed.

One formula for several methods

The paper also writes SFT, rejection-sampling fine-tuning, its online variant, DPO, PPO and GRPO as one gradient: a per-token coefficient times ∇log π_θ. SFT's coefficient is always 1; rejection sampling's is 1 for a correct answer and 0 otherwise; GRPO's is the group-normalized advantage plus a KL term. The methods differ in where the data comes from (a fixed set or fresh samples from the current policy), which reward they use, and how the coefficient is computed.

• G — samples per question, 64 in the paper; 1/|oi| averages over each answer's own length, so every answer carries equal weight whatever its length.
• Âi,t — the group-normalized advantage: (r_i − mean)/std for outcome rewards, or a sum over the current and later steps for process rewards.
• ρi,t, ε — the per-token probability ratio and PPO's clip range.
• β = 0.04 — weight of the KL term. The estimator x − log x − 1, with x the reference-to-policy probability ratio, is zero when the two agree and positive otherwise.

At the extremes: if all G answers get the same reward, every advantage is zero and the question teaches nothing; with G = 1 there is no group to compare against. Later analyses, notably the Dr. GRPO paper, argued that the 1/|o_i| length average and the division by the group's standard deviation introduce biases toward particular response lengths and question difficulties.

The headline results are 7B models with chain-of-thought prompting and no external tools unless noted. GRPO started from DeepSeekMath-Instruct 7B and saw only GSM8K- and MATH-related questions during RL.

With tool-integrated reasoning, MATH went from 57.4% to 58.8%, and self-consistency over 64 samples from DeepSeekMath 7B reached 60.9% on MATH. RL also lifted benchmarks it never trained on, such as the Chinese CMATH, from 84.6% to 88.8%.

The ablations are the most reusable part. On the smaller DeepSeekMath-Instruct 1.3B, process supervision beat outcome supervision under GRPO, which the authors attribute to finer, step-aware credit; rejection-sampling fine-tuning on fresh samples from the current policy pulled ahead of the same method on a fixed set as training went on; and GRPO surpassed that online version. On the 7B model, iterating the reward model helped, most of all in the first iteration.

The negative result reframes the gain. RL raised Maj@K, the accuracy of a majority vote over K samples, but not Pass@K, the chance that any of K samples is right. The authors read this as RL making the output distribution more robust, moving answers the model could already produce to the top, rather than adding fundamentally new ability. One pretraining finding is negative too: arXiv papers, a staple of math corpora, brought no notable improvement on the math benchmarks tested and sometimes made results worse, a conclusion the authors flag as limited to their setup.

The precondition is the limitation: the method needs a scalar reward you can trust, and the cleanest source of one is a verifier. DeepSeekMath itself used a learned reward model, but where correctness can be checked by a program, as in the later R1 work, the reward is hard to game. Where it cannot — helpfulness, tone, judgement, taste — you are back to a learned reward model and rung 6's arguments apply again. The method did not make reinforcement learning cheap in general; it made it cheap for the class of tasks that come with an oracle.

Later analyses found biases in the normalisation: dividing by the group's spread, and averaging over tokens in particular ways, introduces preferences for longer or shorter responses that nobody intended, and several corrections have been proposed. Worth treating the exact normalisation as an open implementation detail rather than as part of the idea — which is precisely the caution the PPO reproducibility studies established for this whole family.

How it held up: this is the optimiser under the reasoning-model wave, and its significance is larger than its arithmetic. It is the rung where post-training stops being about eliciting what pretraining already put in the model and starts being about buying new capability with reinforcement learning at scale — which is the claim the reasoning ladder is built on, and the reason the compute budget of a frontier model is no longer dominated by pretraining alone.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →