PPO,
explained.

DQN learns what each action is worth and then acts greedily, which requires enumerating the actions to take a maximum over them. Policy-gradient methods skip the value and adjust the policy itself in the direction that increases expected return, so a continuous action space is no harder than a discrete one. The price is a failure mode DQN does not have: because the data is collected by the current policy, one over-large update can move the policy somewhere that collects useless data, and there is no way back — the policy has destroyed the source of its own improvement.

The existing fix was to constrain how far the policy may move per update, measured as a divergence between the old and new action distributions, and to enforce it as a hard constraint. That works and requires second-order information, which makes it awkward to implement and expensive to run. PPO's change is to get almost the same effect with first-order methods and a few lines of arithmetic.

The mechanism, in the units that matter: form the probability ratio between the new and old policy for each action actually taken, clip it to a narrow band around 1 (commonly ±0.2), and optimise the minimum of the clipped and unclipped objectives. Taking the minimum is what makes it work rather than merely bounded — it means the objective stops rewarding you for moving further once you have left the band, but still penalises you for a move that made things worse. Because the update is bounded, the same batch of experience can be reused for several epochs instead of one, which is where the sample efficiency over vanilla policy gradients comes from.

Policy-gradient RL learns by trial: run the current policy, see which actions did better than expected — a positive advantage A — and nudge the policy to make those actions more likely. Simple in principle, treacherous in practice. The advantage is estimated from data collected by the old policy, so it's only valid nearby. Take a big step and the new policy is a stranger to that data: the estimate no longer applies, and the update can send performance off a cliff. Worse, RL is on-policy — the broken policy now collects the next batch, so a single bad step can spiral.

The earlier fix, TRPO, solved this with a hard trust region: constrain each update so the new policy stays within a bounded KL distance of the old one. It works, but it needs heavy second-order optimization that's fiddly to implement and scale. PPO (Schulman et al., 2017) asked: can we get the same "don't step too far" safety with plain first-order gradient descent and a couple of lines of code? The answer — a clipped objective — is the reason PPO became the default RL algorithm, all the way up to the alignment of large language models.

To limit "how far the policy moved," PPO needs to measure it. The tool is the probability ratio: for a state-action pair, r = π_new(a|s) / π_old(a|s) — how much more (or less) likely the new policy is to take an action than the old policy that gathered the data. r = 1 is no change; r > 1 means the new policy now favors the action more; r < 1, less.

This ratio is the lever of the whole update. The surrogate objective — the thing plain policy gradients maximize — is r · A: push the ratio up for good actions (positive A), down for bad ones. But nothing stops r from running to 5 or 0.1 on a single batch, and a ratio far from 1 is exactly the danger sign: the policy has moved so far from the data that A is no longer trustworthy. PPO's entire contribution is a way to cap the reward for moving the ratio too far — to make the objective indifferent to steps that leave the safe neighborhood. That's the clip.

PPO takes the surrogate r · A and pins the ratio inside a band [1−ε, 1+ε] (typically ε = 0.2), then takes the minimum of the clipped and unclipped versions. The min is the clever part — it makes the objective a pessimistic bound that only ever removes the incentive to over-step:

The result is a soft trust region built from a clip instead of a constraint. When the ratio is inside the band, PPO behaves like an ordinary policy gradient. When a good action's ratio tries to climb past 1+ε, the clipped term flattens — the objective stops improving, so the gradient for that sample is zero and the update stops pushing. The min ensures the clip only bites in the "helpful" direction, so it never accidentally encourages a large step. All of this is plain arithmetic on scalars — no KL constraint, no conjugate gradients, no line search. That simplicity, with TRPO-like stability, is the whole point.

The clipped surrogate is a single expression. With ratio r, advantage A, and clip range ε:

At r = 1 it's just A. For A > 0, a large r is capped at (1+ε)·A. The min makes it pessimistic — it can only lower the objective relative to r·A, never raise it, so over-stepping is never rewarded.

Worked: with ε = 0.2, a good action whose ratio jumped to 1.5 is capped — but a bad action whose ratio rose to 1.5 is penalized in full, because the min picks the worse value:

Good action, big step up: capped at 1.2 (no runaway reward). Bad action, ratio drifted up: the full −1.5 penalty applies — the clip protects against over-eager increases, not against paying for mistakes. The runnable version below computes the ratio, the clipped objective, and when the ratio leaves the trust region.

PPO is an outer loop around a clipped loss. The loop matters as much as the loss, because the clip is what makes it safe to take more than one gradient step on a batch of fresh experience.

1. Collect with N actors. Each of N parallel copies of the environment runs the current policy for T steps (T is much shorter than an episode), recording states, actions, rewards, the probability the policy gave each action, and the value network's estimate of each state. That gives a batch of N×T steps.
2. Estimate advantages with GAE. For each step compute the one-step error δ_t = R_t + γV(s_t+1) − V(s_t), where R_t is the reward. The advantage is a decaying sum of these errors going forward in time: Â_t = δ_t + (γλ)δ_t+1 + (γλ)^²δ_t+2 + … up to the end of the T-step window. With λ = 0 it uses a single step: low variance, but biased by whatever error the value network has. With λ = 1 it adds up the actual rewards: unbiased but noisy. The Atari and MuJoCo runs use γ = 0.99 and λ = 0.95.
3. Freeze the old policy. Store the action probabilities under the policy that collected the batch. These are the denominators of the ratio r_t = π_θ(a_t|s_t) / π_old(a_t|s_t). Right after collection, every ratio is exactly 1.
4. Run K epochs of minibatch updates. Shuffle the N×T steps, split them into minibatches, and take Adam steps on the combined loss for K passes over the same data. The MuJoCo runs use K = 10 with minibatches of 64 and T = 2048. The Atari runs use K = 3, T = 128 and 8 actors. Each pass moves the ratios away from 1, and the clip is what stops them drifting far.
5. Combine three terms in one loss. When the policy and value function share a network, the objective adds the clipped policy term, subtracts a squared-error loss for the value function, and adds an entropy bonus that keeps the policy from collapsing onto one action too early.
6. Throw the batch away and repeat. The old policy is replaced by the new one, and fresh data is collected. PPO is still on-policy: data is reused for K epochs, not stored for later.

Why the clip and not a penalty

The paper also tries the obvious alternative: instead of clipping, subtract a penalty proportional to the KL divergence between old and new policies. The penalty weight β is adapted: if the measured KL after an update falls below the target divided by 1.5, halve β; if it rises above the target times 1.5, double β. That version works but needed a target to be chosen and was outscored by plain clipping in the authors' comparison.

What the advantage is doing

The advantage says how much better an action turned out than the value network expected. A positive value pushes that action's probability up. A negative one pushes it down. Because the estimate comes from data collected by the old policy, it is only trustworthy while the new policy stays close, which is exactly what the clip protects.

• L^CLIP — the policy term from the ratio r and advantage Â; ε is typically 0.2.
• L^VF — squared error between the value network and its target return; the minus sign makes it a loss when the objective is maximized.
• S[π] — the entropy of the policy at that state; the bonus discourages premature certainty.
• c1, c2 — weights; the Atari runs use 1 and 0.01.
• δt — one-step error from the value network; GAE sums these with weights (γλ)^^k.

Worked example: with ε = 0.2 and a ratio of 1.5, a good action (Â = +1) scores min(1.5, 1.2) = 1.2, so there is no extra reward for going further. A bad action (Â = −1) scores min(−1.5, −1.2) = −1.5, so the full penalty still applies.

The main ablation compared surrogate objectives on seven simulated continuous-control tasks, three random seeds each, with the scores scaled so a random policy is 0 and the best result in the experiment is 1. Averaged over those runs:

Two readings follow. The unconstrained version is worse than random on average, so the constraint is doing real work. And clipping at 0.2 beat every KL-penalty variant the authors tried, fixed or adaptive, even though the penalty is the more principled version.

On the continuous-control tasks over one million timesteps, PPO with clipping matched or beat the earlier trust-region method TRPO, the cross-entropy method, a vanilla policy gradient with adaptive step size, A2C, and A2C with a trust region on most tasks. On 49 Atari games trained for 40 million frames, PPO was compared with A2C and ACER. Counting games by average reward over all of training, PPO won 30, ACER 18 and A2C 1. Counting by the last 100 episodes, ACER won 28, PPO 19, A2C 1 and one game was a tie. So PPO learned faster, while the sample-efficient off-policy method often ended slightly higher.

The paper presents the clip as a practical surrogate, not a proof: it does not guarantee the policy stays within a trust region, it only removes the incentive to leave one. That distinction is worth holding on to, because the failure mode it is defending against has not been eliminated, merely made unlikely under normal settings.

The larger finding came from outside. Reproducibility studies — Engstrom and colleagues, and Andrychowicz and colleagues, both around 2020 — took PPO apart and found that a substantial share of its measured advantage comes from implementation details rather than the clipped objective: advantage normalisation, observation and reward scaling, value-loss clipping, learning-rate annealing, orthogonal initialisation, the generalised advantage estimator's parameters. This is the most important caveat on this ladder, because it generalises: a reported delta between two algorithms can be a delta between two codebases.

How it held up: it became the default policy-gradient method and then found a second career that dwarfed the first. PPO is the optimiser in reinforcement learning from human feedback — the algorithm turning a reward model into a fine-tuned language model on the post-training ladder — and its cost there, four models in memory and a value network to train, is exactly what its descendant GRPO removes. A 2017 continuous-control paper is load-bearing for chat assistants, which is not a lineage anyone predicted.

PPO's second life is in aligning language models. RLHF — reinforcement learning from human feedback — fine-tunes a model to maximize a reward model's score, and that's an RL problem, optimized with PPO. InstructGPT and the RLHF pipelines that made chat assistants helpful and harmless used PPO as the optimizer, precisely because its clipped objective keeps the model from lurching too far from its previous behavior in any single update — exactly what you want when the "policy" is a giant, hard-won language model you don't dare destabilize.

That role made PPO one of the most consequential algorithms in modern AI, far outside its original control-and-games home. It's also a moving target: newer variants trim its machinery — GRPO drops the learned value network and estimates advantages from groups of sampled outputs, simplifying the RLHF stack that PPO defined, and it's central to reasoning-model training like DeepSeek-R1. But the core idea PPO crystallized endures across all of them: improve the policy from experience, and never let a single update wander outside the region where that experience is still trustworthy.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Reward, policy, search. The line from Atari to a Go engine that was never shown a human game — and the algorithm (PPO) that RLHF later borrowed wholesale.

1. 1DQN2015
2. 2PPO2017
3. 3AlphaGo2016
4. 4AlphaZero2018
5. 5MuZero2020

All 110 papers, in 19 ladders →