Direct Preference
Optimization, explained.

The reason a raw language model becomes a helpful assistant is alignment — tuning it to prefer the responses humans actually like. The dominant recipe, RLHF (reinforcement learning from human feedback), does this in three stages: collect human preferences (this answer is better than that one), train a separate reward model to imitate those judgments, then use reinforcement learning (PPO) to push the model toward high-reward outputs.

It works — it's how ChatGPT was aligned — but that third stage is a beast. Reinforcement learning on a giant model is memory-hungry, hyperparameter-sensitive, prone to instability, and happy to "reward-hack" by finding cheap tricks that score high without being good.

DPO's authors did the algebra everyone had skipped. They showed that the reward function RLHF works so hard to learn can be rewritten in terms of the model's own probabilities relative to a frozen reference copy. The policy and the reward aren't two separate things to train — they're two views of the same object.

The paper's memorable framing: your language model is secretly a reward model. If that's true, you never need to build the reward model or run RL against it. You can optimize the model directly.

DPO uses the exact same data as RLHF — pairs of a chosen (preferred) and a rejected response for each prompt — but trains with a single, supervised, classification-style loss: raise the model's probability of the chosen response and lower the rejected one, gently anchored to the reference model so it doesn't drift.

Same inputs, same goal, two stages deleted.

PPO is online reinforcement learning: it samples fresh outputs from the model, scores them, updates, and repeats — a feedback loop with many ways to wobble. DPO is offline and supervised: a fixed dataset of preference pairs, one loss, gradient descent. It trains as calmly and reproducibly as ordinary fine-tuning, on far less hardware.

Across sentiment control, summarization, and single-turn dialogue, DPO matched or beat PPO-based RLHF on how well outputs aligned with human preferences — while being dramatically simpler to implement and train, and far more stable.

DPO became the default alignment method for a huge fraction of open models — its simplicity made preference tuning something any team could do. It also opened a whole family of direct-preference variants (IPO, KTO, ORPO and more), each tweaking the loss for different data or goals.

More deeply, it's a lesson in looking harder before reaching for heavy machinery: an entire reinforcement-learning stage turned out to be unnecessary once someone did the math. Sometimes the reward model you're training is already sitting inside the model you have.