PPO,
explained.

Policy-gradient RL learns by trial: run the current policy, see which actions did better than expected — a positive advantage A — and nudge the policy to make those actions more likely. Simple in principle, treacherous in practice. The advantage is estimated from data collected by the old policy, so it's only valid nearby. Take a big step and the new policy is a stranger to that data: the estimate no longer applies, and the update can send performance off a cliff. Worse, RL is on-policy — the broken policy now collects the next batch, so a single bad step can spiral.

The earlier fix, TRPO, solved this with a hard trust region: constrain each update so the new policy stays within a bounded KL distance of the old one. It works, but it needs heavy second-order optimization that's fiddly to implement and scale. PPO (Schulman et al., 2017) asked: can we get the same "don't step too far" safety with plain first-order gradient descent and a couple of lines of code? The answer — a clipped objective — is the reason PPO became the default RL algorithm, all the way up to the alignment of large language models.

To limit "how far the policy moved," PPO needs to measure it. The tool is the probability ratio: for a state-action pair, r = π_new(a|s) / π_old(a|s) — how much more (or less) likely the new policy is to take an action than the old policy that gathered the data. r = 1 is no change; r > 1 means the new policy now favors the action more; r < 1, less.

This ratio is the lever of the whole update. The surrogate objective — the thing plain policy gradients maximize — is r · A: push the ratio up for good actions (positive A), down for bad ones. But nothing stops r from running to 5 or 0.1 on a single batch, and a ratio far from 1 is exactly the danger sign: the policy has moved so far from the data that A is no longer trustworthy. PPO's entire contribution is a way to cap the reward for moving the ratio too far — to make the objective indifferent to steps that leave the safe neighborhood. That's the clip.

PPO takes the surrogate r · A and pins the ratio inside a band [1−ε, 1+ε] (typically ε = 0.2), then takes the minimum of the clipped and unclipped versions. The min is the clever part — it makes the objective a pessimistic bound that only ever removes the incentive to over-step:

The result is a soft trust region built from a clip instead of a constraint. When the ratio is inside the band, PPO behaves like an ordinary policy gradient. When a good action's ratio tries to climb past 1+ε, the clipped term flattens — the objective stops improving, so the gradient for that sample is zero and the update stops pushing. The min ensures the clip only bites in the "helpful" direction, so it never accidentally encourages a large step. All of this is plain arithmetic on scalars — no KL constraint, no conjugate gradients, no line search. That simplicity, with TRPO-like stability, is the whole point.

The clipped surrogate is a single expression. With ratio r, advantage A, and clip range ε:

At r = 1 it's just A. For A > 0, a large r is capped at (1+ε)·A. The min makes it pessimistic — it can only lower the objective relative to r·A, never raise it, so over-stepping is never rewarded.

Worked: with ε = 0.2, a good action whose ratio jumped to 1.5 is capped — but a bad action whose ratio rose to 1.5 is penalized in full, because the min picks the worse value:

Good action, big step up: capped at 1.2 (no runaway reward). Bad action, ratio drifted up: the full −1.5 penalty applies — the clip protects against over-eager increases, not against paying for mistakes. The runnable version below computes the ratio, the clipped objective, and when the ratio leaves the trust region.

PPO delivered TRPO-level stability with first-order simplicity, and became the RL default:

PPO isn't the most sample-efficient RL algorithm, and off-policy methods can beat it on that axis. But its combination of stability, simplicity, and robustness made it the pragmatic first choice for a huge range of problems — the algorithm you reach for when you want something that reliably works rather than something exotic.

PPO's second life is in aligning language models. RLHF — reinforcement learning from human feedback — fine-tunes a model to maximize a reward model's score, and that's an RL problem, optimized with PPO. InstructGPT and the RLHF pipelines that made chat assistants helpful and harmless used PPO as the optimizer, precisely because its clipped objective keeps the model from lurching too far from its previous behavior in any single update — exactly what you want when the "policy" is a giant, hard-won language model you don't dare destabilize.

That role made PPO one of the most consequential algorithms in modern AI, far outside its original control-and-games home. It's also a moving target: newer variants trim its machinery — GRPO drops the learned value network and estimates advantages from groups of sampled outputs, simplifying the RLHF stack that PPO defined, and it's central to reasoning-model training like DeepSeek-R1. But the core idea PPO crystallized endures across all of them: improve the policy from experience, and never let a single update wander outside the region where that experience is still trustworthy.