GRPO,
explained.

Reinforcement learning is how you teach a model from outcomes instead of examples: generate an answer, score it, nudge the weights toward what scored well. The catch is the nudging. A policy gradient needs to know not just "was this good?" but "was this better than expected?" — otherwise every sample looks equally worth reinforcing and training is hopelessly noisy.

"Better than expected" requires a baseline: an estimate of the expected reward. The standard answer — PPO, inherited from InstructGPT and the RLHF lineage — learns that baseline with a value network (the critic). For LLMs, the critic is typically another copy of the model. Training one 7B policy means holding a 7B critic beside it, training both, syncing both. The bill doubles before you generate a single token.

Four networks sit in memory during classic RLHF-style PPO: the policy being trained, the frozen reference model (to keep outputs from drifting too far), the reward model (what scores answers), and the critic (what predicts expected score, token by token). The critic exists purely to compute the advantage — how much better this output was than the expectation.

The critic is also the fragile one: it must learn to predict rewards for partial generations — a moving target that shifts every time the policy improves. A mis-trained critic silently poisons the advantages, and with them, the whole run.

Group Relative Policy Optimization deletes the critic with one observation: if you sample several answers to the same prompt, you already have an estimate of the expected reward — the group's average. No learned predictor needed; the cohort grades itself.

Each answer's advantage is simply its reward minus the group mean (normalized by the group's spread). Beat your siblings → reinforced. Trail them → suppressed. A KL penalty against the reference model still keeps the policy from wandering off. The result is PPO's learning signal with roughly half the memory footprint and none of the critic's training instability.

GRPO's sweet spot is domains with verifiable rewards: math answers check, code runs or doesn't, proofs verify. The signal is crisp and ungameable, so the group comparison measures something real. This pairing — group-relative advantages plus checkable correctness — is what later let reasoning be incentivized into existence rather than imitated.

The within-group comparison is also exactly the right granularity. An absolute score can't distinguish "hard problem, decent attempt" from "easy problem, sloppy attempt." Comparing attempts at the same prompt cancels problem difficulty out — the model learns which of its own strategies works, prompt by prompt. It's a tournament among siblings, not a grade against an absolute curve.

The paper's vehicle was DeepSeekMath 7B. Step one had nothing to do with RL: continue pretraining a code model on ~120B tokens of carefully filtered math scraped from the web — a data-curation feat in itself. Step two: instruction-tune. Step three: GRPO.

A 7B open model approaching the math performance of vastly larger closed models was the headline. The quieter, more consequential result was the recipe: quality data plus cheap, stable RL substitutes for scale.

A year later, DeepSeek pointed GRPO at a stronger base model, kept only the correctness reward, and skipped supervised reasoning data entirely. The result was DeepSeek-R1-Zero — chain-of-thought reasoning, self-checking, the famous "aha moment," all emerging from GRPO pressure toward correct answers. GRPO wasn't a footnote in that story; it was the engine that made RL cheap and stable enough to run at that scale.

The broader field followed. Group-relative, critic-free RL became the default recipe for reasoning training across open labs, spawning a family of refinements, and "RL with verifiable rewards" became the standard phrase for the whole paradigm — the training-side twin of test-time compute.

GRPO inherits RL's oldest failure mode: reward hacking. If the reward is even slightly gameable — a weak checker, a length bias, a format loophole — group pressure will find and exploit it ruthlessly. It also leans on cheap sampling: every training prompt costs G full generations, trading the critic's memory for extra inference compute.

And it shares outcome-based RL's boundary: where correctness isn't checkable — taste, judgment, open-ended writing — the crisp signal disappears, and you're back to learned reward models with all their softness. GRPO didn't solve alignment for fuzzy domains; it made RL practical where the answer key exists.