GRPO
also: group relative policy optimization
The RL method behind DeepSeek-R1: score groups of sampled answers relative to each other, no value network.
GRPO samples a group of answers per prompt, scores them (e.g. correct/incorrect), and updates the policy toward answers that beat the group average — replacing PPO’s learned value network with a simple group baseline. Cheap enough to run at scale on verifiable tasks (math, code), it is how DeepSeek-R1 learned o1-style reasoning from a base model with no supervised warm-up.
Worked example: group relative policy optimization: an RL method that, instead of training a separate value/critic model, samples a GROUP of answers to the same prompt and uses their average reward as the baseline — an answer above its group’s average is reinforced, below is discouraged. Gotcha: dropping the critic makes it simpler and cheaper than PPO (no second network to train), but it needs several samples per prompt (more rollouts) and a reward signal that meaningfully varies within a group; it became widely known as a method behind reasoning-model RL training.