Direct Preference Optimization skips the reward model and the RL loop: it optimizes the policy directly on preference pairs (chosen vs rejected) with a closed-form loss that provably targets the same objective. Stabler and far simpler to run than PPO-based RLHF, it became the default alignment recipe at most labs.
Worked example: given a chosen and a rejected response per prompt, DPO directly raises the chosen’s likelihood and lowers the rejected’s (relative to a frozen reference) in one supervised-style loss — no reward model, no RL loop. Gotcha: it is only as good as the preference data’s quality and coverage, and can over-optimize on narrow pairs; it trades some of online RL’s flexibility for a lot of stability.