GRPO,
explained.

Reinforcement learning is how you teach a model from outcomes instead of examples: generate an answer, score it, nudge the weights toward what scored well. The catch is the nudging. A policy gradient needs to know not just "was this good?" but "was this better than expected?" — otherwise every sample looks equally worth reinforcing and training is hopelessly noisy.

"Better than expected" requires a baseline: an estimate of the expected reward. The standard answer — PPO, inherited from InstructGPT and the RLHF lineage — learns that baseline with a value network (the critic). For LLMs, the critic is typically another copy of the model. Training one 7B policy means holding a 7B critic beside it, training both, syncing both. The bill doubles before you generate a single token.

Four networks sit in memory during classic RLHF-style PPO: the policy being trained, the frozen reference model (to keep outputs from drifting too far), the reward model (what scores answers), and the critic (what predicts expected score, token by token). The critic exists purely to compute the advantage — how much better this output was than the expectation.

The critic is also the fragile one: it must learn to predict rewards for partial generations — a moving target that shifts every time the policy improves. A mis-trained critic silently poisons the advantages, and with them, the whole run.

Group Relative Policy Optimization deletes the critic with one observation: if you sample several answers to the same prompt, you already have an estimate of the expected reward — the group's average. No learned predictor needed; the cohort grades itself.

Each answer's advantage is simply its reward minus the group mean (normalized by the group's spread). Beat your siblings → reinforced. Trail them → suppressed. A KL penalty against the reference model still keeps the policy from wandering off. The result is PPO's learning signal with roughly half the memory footprint and none of the critic's training instability.

The whole advantage computation is one line — no network, just the group's own statistics:

Two properties fall straight out of that formula. Subtracting the mean makes the advantages sum to zero — every step reinforces the above-average answers exactly as much as it suppresses the below-average ones, a built-in baseline. And dividing by the group's spread means an answer is judged only against its own siblings, so problem difficulty cancels: a mediocre answer to a brutal problem can still be reinforced if it beat the pack. The runnable version below shows the same 0.5 reward earning a positive advantage on a hard prompt and a negative one on an easy prompt.

GRPO's sweet spot is domains with verifiable rewards: math answers check, code runs or doesn't, proofs verify. The signal is crisp and ungameable, so the group comparison measures something real. This pairing — group-relative advantages plus checkable correctness — is what later let reasoning be incentivized into existence rather than imitated.

The within-group comparison is also exactly the right granularity. An absolute score can't distinguish "hard problem, decent attempt" from "easy problem, sloppy attempt." Comparing attempts at the same prompt cancels problem difficulty out — the model learns which of its own strategies works, prompt by prompt. It's a tournament among siblings, not a grade against an absolute curve.

The paper's vehicle was DeepSeekMath 7B. Step one had nothing to do with RL: continue pretraining a code model on ~120B tokens of carefully filtered math scraped from the web — a data-curation feat in itself. Step two: instruction-tune. Step three: GRPO.

A 7B open model approaching the math performance of vastly larger closed models was the headline. The quieter, more consequential result was the recipe: quality data plus cheap, stable RL substitutes for scale.

A year later, DeepSeek pointed GRPO at a stronger base model, kept only the correctness reward, and skipped supervised reasoning data entirely. The result was DeepSeek-R1-Zero — chain-of-thought reasoning, self-checking, the famous "aha moment," all emerging from GRPO pressure toward correct answers. GRPO wasn't a footnote in that story; it was the engine that made RL cheap and stable enough to run at that scale.

The broader field followed. Group-relative, critic-free RL became the default recipe for reasoning training across open labs, spawning a family of refinements, and "RL with verifiable rewards" became the standard phrase for the whole paradigm — the training-side twin of test-time compute.

GRPO inherits RL's oldest failure mode: reward hacking. If the reward is even slightly gameable — a weak checker, a length bias, a format loophole — group pressure will find and exploit it ruthlessly. It also leans on cheap sampling: every training prompt costs G full generations, trading the critic's memory for extra inference compute.

And it shares outcome-based RL's boundary: where correctness isn't checkable — taste, judgment, open-ended writing — the crisp signal disappears, and you're back to learned reward models with all their softness. GRPO didn't solve alignment for fuzzy domains; it made RL practical where the answer key exists.

GRPO (group relative policy optimization) is a reinforcement-learning method that drops PPO separate value/critic network: instead it samples a GROUP of responses per prompt and uses the group average reward as the baseline, reinforcing responses above average and discouraging those below.

It mattered because removing the critic makes RL training simpler and cheaper — no second network the size of the model to train — and it became widely associated with training strong reasoning models via reinforcement learning.

It shows up in reasoning-model RL pipelines and as a lighter-weight alternative to PPO when you can afford several samples per prompt and have a usable reward signal.

The lasting lesson is that you can replace a learned baseline with a cheap empirical one (the group mean) and keep the useful part of policy-gradient RL — simplifying the machinery can matter as much as scaling it. It ties to PPO, RLHF, rollouts, and test-time compute.