Reinforcement Learning from Human Feedback aligns a language model to human preferences in three steps: collect comparisons where people pick the better of two model responses, train a reward model to predict those preferences as a score, then optimize the language model (the policy) with reinforcement learning to maximize that reward — with a KL penalty keeping it close to the original model. It is how base models became helpful assistants.

What is reward hacking?

The reward model is only an imperfect proxy for human preference. If the policy optimizes it too hard, it finds ways to score high that humans don't actually like — excessive length, sycophancy, confident-sounding filler — exploiting gaps in the reward model rather than genuinely improving. A KL penalty that anchors the policy to the original model is the main defense.

What is DPO and how does it differ from RLHF?

Direct Preference Optimization trains a model directly on preference pairs with a simple classification-style loss, skipping the separate reward model and the reinforcement-learning loop. A clever derivation shows the language model can implicitly serve as its own reward model, so DPO targets the same objective as RLHF but is far simpler, cheaper and more stable to train — which is why it became the common default.

Why not just fine-tune on good answers instead of preferences?

Supervised fine-tuning on good answers teaches the model to imitate examples, but it can't express that answer A is better than answer B, or by how much. Preference data captures relative quality and subtle trade-offs (helpful but not harmful, confident but not sycophantic) that are hard to demonstrate directly, letting alignment shape behavior beyond simple imitation.

LABS · 21 / ALIGNMENT

The Taste Trainer.

A base model can write, but it doesn't know what people prefer. So we teach it taste: show humans two answers, let them pick the better one, thousands of times. Train a reward model on those picks, then push the model to chase that reward — and watch it start to cheat. Then see how DPO skips the whole detour.

Act 1 · Preferences → reward model

Pick the better answer

You are the human labeler. For each prompt, choose the response you prefer. Every pick is a data point; a reward model learns to predict which answer people like — turning fuzzy human taste into a number the machine can optimize.

Prompt: "Explain photosynthesis to a 10-year-old."

Comparisons labeled

—

Reward model accuracy

predicting your taste

—

Your revealed preference

Each comparison sharpens the reward model's picture of what you value. After enough picks it can score a brand-new answer the way you would — that score is the training signal for the next stage.

Act 2 · Policy optimization (and its trap)

The model chases the reward — too hard

Now the model (the policy) is optimized to maximize the reward model's score. Turn up the optimization pressure and watch reward climb — but the reward model is only a proxy. Push too far and the policy finds cheap tricks that score high but humans hate: reward hacking. A KL penalty anchors it back.

Optimization pressure50%

KL penalty (stay close to base)medium

—

Reward-model score

the proxy

—

True human quality

—

Hacking gap

Watch the two curves diverge: proxy reward keeps rising while true quality peaks then falls. The gap is reward hacking — sycophancy, length-padding, confident filler. The KL penalty is what keeps the policy honest.

Act 3 · The shortcut

DPO: skip the reward model entirely

RLHF is a three-stage pipeline: label → train reward model → RL loop. DPO proved you can train straight from the preference pairs with a simple classification loss — the model implicitly is its own reward model. Same goal, far less machinery. Toggle between them.

Method:

Pipeline stages

Models to train

reward + policy

medium

Training stability

DPO collapses three stages into one and one loss, targeting the same objective RLHF does — which is why it became the common default for preference tuning. RLHF still wins where you need an explicit, reusable reward model or online RL.

Teaching a model what "good" means

A pretrained model has read the internet; a fine-tuned one can follow instructions. Neither knows what people actually prefer — helpful over evasive, honest over confident-sounding, concise over padded. Those are relative judgments, hard to demonstrate directly. So instead of showing the model answers to copy, we show it comparisons: this response is better than that one. That's the raw material of alignment.

The RLHF pipeline

Step 1 — collect preferences: humans pick the better of two model responses, thousands of times. Step 2 — train a reward model: a model learns to predict those picks, outputting a scalar "how much would a human like this?" score. Step 3 — optimize the policy: reinforcement learning (classically PPO) pushes the language model to produce responses the reward model scores highly.

The reward-hacking trap

The reward model is a proxy, not the real thing — and optimizers exploit proxies. Push too hard and the policy discovers tricks that inflate the score without genuinely being better: sycophancy, verbosity, confident filler. The fix is a KL penalty that keeps the policy close to the original model, trading a little reward for staying grounded. It's Goodhart's law in a training loop: when a measure becomes a target, it stops measuring well.

DPO: the elegant shortcut

Direct Preference Optimization asked: do we even need the separate reward model and RL loop? Its derivation shows the language model can implicitly act as its own reward model, so you can train directly on preference pairs with a simple classification-style loss — no reward model, no PPO, no sampling loop. Same underlying objective as RLHF, dramatically simpler and more stable, which is why it became the common default for preference tuning.

And beyond

The family keeps growing: Constitutional AI replaces human preference labels with AI judgments against written principles (RLAIF); reasoning models like DeepSeek-R1 use RL against verifiable rewards (is the answer correct?) rather than learned preference. But the core idea traces to InstructGPT: turn human judgment into a signal, and optimize toward it — carefully.

Check yourself

Four questions. No sycophancy, please.

Finished this one? 0 / 22 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

The Taste Trainer.

Teaching a model what "good" means

The RLHF pipeline

DPO: the elegant shortcut

And beyond

Check yourself

Explore the topic

More Labs

Explore more from Vibe Engines

Teaching a model what "good" means

The RLHF pipeline

DPO: the elegant shortcut

And beyond

Check yourself

Keep going

InstructGPT (RLHF) →

DPO →

Constitutional AI →

Cross-Entropy →

Explore the topic

More Labs

Explore more from Vibe Engines

Get the next one in your inbox.