Direct Preference
Optimization, explained.

Rung 1's step three is the expensive and fragile one: train a separate reward model, then run reinforcement learning against it while holding four models in play and watching for divergence. This rung removes the reward model and the reinforcement learning, and it does so by algebra rather than approximation — which is why it is a different kind of result from most of this ladder.

The argument, in outline. The objective rung 1 optimises is "maximise reward while staying close to the reference model", and for that objective the optimal policy has a closed form in terms of the reward. Because the relation is invertible, the reward can equally be written in terms of the policy — so a preference between two responses can be expressed directly as a function of the policy's own probabilities. The reward model does not need to be built, because the language model already implies one. What is left is a supervised loss that raises the probability of the preferred response relative to the rejected one, weighted so that pairs the model already gets right contribute little.

Stated in rung 1's units: four models and an RL loop become two models and a classification-style loss — the policy being trained and a frozen reference. No reward model to train, no value network, no sampling during training, no explicit reward model to over-exploit (though DPO can still over-optimize against the preference data, as later work showed), and the failure modes are closer to the familiar ones of supervised learning.

The reason a raw language model becomes a helpful assistant is alignment — tuning it to prefer the responses humans actually like. The dominant recipe, RLHF (reinforcement learning from human feedback), does this in three stages: collect human preferences (this answer is better than that one), train a separate reward model to imitate those judgments, then use reinforcement learning (PPO) to push the model toward high-reward outputs.

It works — it's how ChatGPT was aligned — but that third stage is a beast. Reinforcement learning on a giant model is memory-hungry, hyperparameter-sensitive, prone to instability, and happy to "reward-hack" by finding cheap tricks that score high without being good.

DPO's authors did the algebra everyone had skipped. They showed that the reward function RLHF works so hard to learn can be rewritten in terms of the model's own probabilities relative to a frozen reference copy. The policy and the reward aren't two separate things to train — they're two views of the same object.

The paper's memorable framing: your language model is secretly a reward model. If that's true, you never need to build the reward model or run RL against it. You can optimize the model directly.

DPO uses the exact same data as RLHF — pairs of a chosen (preferred) and a rejected response for each prompt — but trains with a single, supervised, classification-style loss: raise the model's probability of the chosen response and lower the rejected one, gently anchored to the reference model so it doesn't drift.

Same inputs, same goal, two stages deleted.

Here is the entire method as one line. For a prompt x with chosen response y_w and rejected response y_l:

Each bracketed term is a log-ratio: how much more (or less) the policy likes a response than the frozen reference does. The paper's key claim is that β times this log-ratio is the reward — the reward model RLHF trains separately already lives implicitly inside the policy. Concrete numbers, with β = 0.1:

Margin = 2 − (−2) = 4, so the loss is −log σ(0.1 × 4) = −log σ(0.4) = −log 0.5987 = 0.513. Three behaviors fall straight out of the formula:

That last column is DPO's quiet elegance: the gradient on each pair is scaled by σ(−βΔ) — how wrong the model currently is about that pair. Examples it already ranks correctly fade; examples it ranks backwards shout. And β controls the anchor: small β lets the policy drift far from the reference, large β pins it close — the same KL-leash PPO enforces with machinery, achieved by one scalar.

PPO is online reinforcement learning: it samples fresh outputs from the model, scores them, updates, and repeats — a feedback loop with many ways to wobble. DPO is offline and supervised: a fixed dataset of preference pairs, one loss, gradient descent. It trains as calmly and reproducibly as ordinary fine-tuning, on far less hardware.

DPO uses exactly the data RLHF uses: prompts x, each with a preferred response y_w and a dispreferred response y_l. The difference is that the RLHF objective is solved on paper first, and only the result is trained. The derivation is short enough to follow line by line.

1. Start from the RLHF objective. Maximize expected reward while staying close to a reference policy: max_π E_{x, y ~ π}[r(x, y)] − β·D_KL[π(y | x) ‖ π_ref(y | x)]. This is the objective PPO approximately optimizes in InstructGPT-style pipelines.
2. Solve it in closed form. For any reward function, the maximizer is π*(y | x) = π_ref(y | x)·exp(r(x, y)/β) / Z(x), where Z(x) = Σ_y π_ref(y | x)·exp(r(x, y)/β). Z sums over every possible response, so it cannot be computed, which is why nobody samples from this formula directly.
3. Invert for the reward. Taking logs and rearranging gives r(x, y) = β·log[π*(y | x)/π_ref(y | x)] + β·log Z(x). Any reward can be written through the policy it induces, plus a term that depends only on the prompt.
4. Substitute into Bradley–Terry. The standard preference model is p(y_w ≻ y_l | x) = σ(r(x, y_w) − r(x, y_l)). Both responses share the prompt, so β·log Z(x) appears in both rewards and cancels in the difference. The preference probability now depends only on log-ratios of the policy to the reference.
5. Fit by maximum likelihood. Replace π* with the trainable π_θ and minimize the negative log-likelihood of the observed preferences. That is the DPO loss in the worked example above, and its implicit reward is r̂_θ(x, y) = β·log[π_θ(y | x)/π_ref(y | x)].
6. Train it like supervised learning. Set π_ref to the supervised fine-tuned model. When none exists, the paper first fine-tunes on the preferred responses (Preferred-FT) and uses that as the reference, so the reference sits close to the data. Each step needs the log-probabilities of y_w and y_l, summed over their response tokens with the prompt excluded, under the policy and under the frozen reference; the reference values can be computed once, ahead of time. Nothing is sampled during training.

Why the shortcut loses nothing

Rewards that differ only by a function of the prompt induce the same preferences and the same optimal policy, so they form an equivalence class. The paper proves that every such class consistent with the Bradley–Terry model, and with its ranking generalization Plackett–Luce, contains a reward of the form β·log[π(y | x)/π_ref(y | x)] for some policy π. Writing the reward through the policy therefore does not restrict which preferences can be learned.

Settings used in the paper

The default was β = 0.1 with batch size 64 and the RMSprop optimizer at a learning rate of 10^⁻⁶, warmed up linearly over 150 steps; the summarization experiments used β = 0.5. At β = 0.1, a pair's implicit reward gap is one tenth of its log-ratio margin.

Differentiating the DPO loss gives:

• ∇ log πθ(yw | x) — raises the probability of the preferred response.
• − ∇ log πθ(yl | x) — lowers the probability of the dispreferred one.
• σ(r̂l − r̂w) — the per-pair weight: large when the implicit reward ranks the pair the wrong way round, small when it already ranks it correctly.
• β — the KL strength inherited from the RLHF objective; it scales both the implicit reward and the gradient.

The weight is not decoration. The paper reports that a naive version without it, essentially unlikelihood training, made the model degenerate; on summarization it produced long runs of a single repeated word. At the extremes of β: as β grows, a small change in log-ratio already saturates the sigmoid, so the policy is held close to π_ref; as β shrinks, fitting the same preferences requires much larger log-ratios, which means drifting further from the reference.

Three tasks were chosen so that one has a known true reward. In controlled sentiment generation, GPT-2-large continues 2–8-token prefixes of IMDb reviews and should write positively; preferences come from a sentiment classifier that also serves as the ground-truth reward. In summarization, a GPT-J model fine-tuned on Reddit TL;DR summaries learns from human preference data. In single-turn dialogue, Pythia-2.8B learns from Anthropic's helpful-and-harmless (HH) preferences. Baselines included PPO with a learned reward model, PPO with the ground-truth reward (PPO-GT, sentiment only), Preferred-FT, unlikelihood training and best-of-N sampling. GPT-4 judged summaries and dialogue.

A human study checked the judge: GPT-4's verdicts agreed with human raters about as often as the raters agreed with each other. DPO was also more robust to sampling temperature, while PPO's summaries degraded toward the base GPT-J's quality at high temperature.

The negative results came from the simpler baselines. Unlikelihood training, which pushes the preferred response up and the rejected one down with no per-pair weighting, degenerated on summarization and dialogue, producing meaningless text, and was left out of those experiments. And the PPO model tried for dialogue could not improve on the base Pythia-2.8B at all.

The structural limitation is that the method is offline. It learns from a fixed set of preference pairs and never generates a new response to be judged, whereas rung 1's loop samples from the current policy and scores what it actually produces — so it keeps getting signal about the model's present behaviour rather than a previous model's. When there is budget to collect fresh judgements, several careful comparisons found reinforcement learning with a reward model still ahead, and the honest summary of the field's position is a conditional: take this rung for simplicity and a fixed dataset, take rung 1's shape when you can afford to keep scoring new samples.

Two practical failure modes were documented afterwards. Results are sensitive to the strength of the reference constraint, and the loss can reduce the probability of both responses in a pair while still improving their ratio — it optimises a relative quantity, and a relative quantity can be satisfied in ways nobody wanted.

How it held up: enormously adopted, and then largely bypassed at the frontier for a reason that has little to do with its merits. Its whole premise is that the reward must be learned from preferences, which is true for helpfulness and tone. For mathematics and code the reward can be computed — run the tests — and once the reward is a program rather than a model, the objections to reinforcement learning largely dissolve, which is the opening the next rung walks through.

DPO became the default alignment method for a huge fraction of open models — its simplicity made preference tuning something any team could do. It also opened a whole family of direct-preference variants (IPO, KTO, ORPO and more), each tweaking the loss for different data or goals.

More deeply, it's a lesson in looking harder before reaching for heavy machinery: an entire reinforcement-learning stage turned out to be unnecessary once someone did the math. Sometimes the reward model you're training is already sitting inside the model you have.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →