The big idea

What does RLHF do?

A pretrained base model is astonishingly capable — it has read the internet — but it's not aligned: it predicts plausible text, not helpful, honest, harmless answers. Ask it a question and it might continue with more questions, ramble, or produce unsafe content. The gap between "capable text predictor" and "assistant that does what people want" is what alignment closes.

RLHF (reinforcement learning from human feedback) aligns a base model to human preferences in three stages: supervised fine-tuning on demonstrations (teach it to follow instructions), a reward model trained on human comparisons (learn what people prefer), and RL optimization (PPO) against that reward — with a KL constraint so it improves without going off the rails.

How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the pipeline grow, and at the end cut the KL leash and degrade the preference data to see reward hacking and garbage-in. Hit Begin.

Step 1 · Capable but unaligned

Why pretraining isn't enough

The base model has read everything and can generate fluent text on anything. So why isn't it already a good assistant — and why can't you just prompt your way to alignment?

Design decision: A base model is hugely capable. Why isn't it already an aligned assistant?

The call: It doesn't know enough facts. — Knowledge isn't the gap — the base model knows a great deal. The gap is behavior/objective: it predicts plausible text rather than being a helpful, safe assistant, which alignment training instills.

Pretraining optimizes next-token prediction over internet text, so the base model learned to mimic the distribution of text — including rambling, refusing to answer, or unsafe continuations — not to be helpful, honest, and harmless. That's a different objective from "assist the user." Prompting nudges it, but you can't reliably prompt-engineer alignment into a model; you have to train it toward human intent. That training is RLHF.

Capability ≠ alignment: Pretraining buys capability (it knows and can generate). Alignment buys behavior (it does what people want, safely). They're separate: a capable model can be unaligned, and alignment is a distinct training phase that reshapes behavior toward human preferences.

Step 2 · Three stages

The alignment pipeline

Given a capable-but-unaligned base model, what's the overall recipe to turn it into an aligned assistant?

Three stages on top of the base model. (1) Supervised fine-tuning (SFT): fine-tune on human demonstrations of good answers, teaching instruction-following. (2) Reward model (RM): collect human preference comparisons and train a model to predict which output people prefer. (3) RL optimization: use RL (PPO) to optimize the SFT model to produce outputs the reward model scores highly — kept close to the SFT model by a KL penalty. SFT teaches the format; RM+RL refine toward preference.

SFT → RM → RL: The canonical InstructGPT recipe: SFT gives a decent instruction-follower, the reward model captures nuanced human preference that's hard to demonstrate, and RL pushes the policy up the reward while a KL leash keeps it from drifting into nonsense. Each stage does something the others can't.

Step 3 · Teach the format

Supervised fine-tuning

First, get the base model to actually respond like an assistant. What data and objective do that?

SFT: fine-tune the base model on a dataset of demonstrations — prompts paired with high-quality human-written (or curated) responses — using standard supervised next-token loss. This teaches the model the format and behavior of following instructions: answer the question, in the right style, helpfully. The result is a solid instruction-following model — and, importantly, the starting point and reference for the RL stage.

Imitate good answers: SFT is imitation learning: show the model examples of good responses and it learns to produce that kind of output. It gets you most of the way to a usable assistant. But demonstrations have limits (next step), which is why RLHF adds preference learning on top.

Step 4 · Why not stop at SFT?

Preferences beat demonstrations

SFT already gives a decent assistant. Why add the whole reward-model + RL machinery instead of just collecting more demonstrations?

Design decision: SFT works reasonably. Why add reward-model + RL instead of just more demonstrations?

The call: It's easier and richer for humans to *compare* two outputs than to write the ideal one — comparisons capture nuanced quality that demonstrations can't, at lower cost. — Writing a perfect demonstration is hard and doesn't convey how much better it is than alternatives. Having humans compare two model outputs (A vs B) is faster, cheaper, and captures nuanced preferences (tone, helpfulness, safety) that are hard to demonstrate — giving a reward signal you can then optimize against at scale.

Demonstrations are expensive and limited: writing the single ideal answer is hard, and it doesn't tell the model how much better it is than the alternatives. Humans find it far easier and cheaper to compare two outputs ("A is better than B") than to author the perfect one — and those comparisons capture nuanced quality (tone, helpfulness, safety, subtle correctness) that's hard to demonstrate. Preferences give a richer, scalable signal you can turn into a reward and optimize against.

Compare, don't author: The key insight of RLHF: judging is easier than generating. People can't always write the best answer, but they can reliably rank two answers. Harvesting that comparison signal — and modeling it — unlocks preference optimization beyond what demonstrations alone can teach.

Step 5 · Model the preference

The reward model

You have human comparisons, but RL needs a score for any output on the fly — you can't ask a human on every RL step. How do you turn sparse human comparisons into a dense reward signal?

Train a reward model: take the human preference pairs (for a prompt, output A preferred over B) and train a model to output a scalar score such that preferred outputs score higher (a ranking loss). Now you have a fast, automatic proxy for human preference that can score any output during RL — a dense reward from sparse human labels. The reward model is typically initialized from the SFT model with a scalar head. Its quality is bounded by the preference data quality.

Learn a preference scorer: The reward model generalizes human judgments into a function that scores unseen outputs. It's how you scale rare, expensive human comparisons into a reward usable on millions of RL samples. Crucially it's a proxy — good but imperfect — which is why RL must be constrained (next step).

Step 6 · Optimize, on a leash

RL (PPO) with a KL constraint

Now push the model to produce high-reward outputs. But the reward model is an imperfect proxy — optimize it too hard and the model will find degenerate outputs that score high but are actually bad (reward hacking). How do you optimize without going off the rails?

Design decision: RL against an imperfect reward model can be "hacked". How do you optimize safely?

The call: Just maximize the reward model score as hard as possible. — Unconstrained optimization of an imperfect proxy is exactly reward hacking — the policy finds weird outputs that fool the reward model but are worse to humans. You must constrain it toward the reference.

Use RL — PPO — to optimize the policy to maximize the reward, minus a KL-divergence penalty from the frozen SFT reference model. The reward pulls the policy toward human-preferred outputs; the KL penalty keeps it close to known-good behavior, so it can't drift far to exploit reward-model artifacts. That balance — climb the reward, stay near the reference — is what lets RLHF improve alignment without reward hacking or degenerating into gibberish that games the proxy.

Reward + KL leash: The objective is reward − β·KL(policy ‖ reference). Because the reward model is an imperfect proxy (Goodhart: optimize a proxy too hard and it stops correlating with the goal), the KL term is essential — it bounds how far the policy can move, keeping optimization honest. Tuning β trades alignment gain against drift.

Step 7 · Simpler & cheaper variants

DPO and RLAIF

Full PPO-based RLHF is complex and finicky (a reward model + on-policy RL + a reference, all in the loop). Are there simpler or cheaper ways to get the same alignment?

Two big alternatives. DPO (Direct Preference Optimization) skips the explicit reward model and RL loop: it optimizes the policy directly on the preference pairs with a classification-style loss that provably targets the same objective — much simpler and more stable, now very common. RLAIF (RL from AI Feedback / Constitutional AI) replaces (some) expensive human comparisons with AI-generated preferences guided by a set of principles — scaling the feedback cheaply. Both attack RLHF's costs: DPO its complexity, RLAIF its human-labeling expense.

DPO & RLAIF: DPO: derive the optimal policy directly from preferences, no separate reward model or PPO — simpler, stabler, popular. RLAIF: use an AI (with a constitution/principles) to produce preference labels, cutting human cost. The three-stage PPO recipe is the classic; these are the modern, cheaper paths to the same goal.

Step 8 · The sharp edges

Hacking, overoptimization & labelers

RLHF's hard problems are real: reward hacking, reward-model overoptimization, the quality and bias of human labelers, and evaluating whether the model actually got better.

Contain reward hacking with the KL constraint and by not over-training. Watch overoptimization: the reward model is a proxy, so beyond a point more RL degrades true quality even as reward-model score rises (Goodhart) — monitor real evals, not just reward. Invest in labeler quality: clear guidelines, inter-annotator agreement, and calibration, because the whole pipeline inherits labelers' judgments and biases (whose preferences? is a live question). And evaluate alignment with held-out human evals / red-teaming, not the reward model — the reward is the means, human-judged behavior is the goal.

Design for the unhappy path: Hacking → KL leash + early stop. Overoptimization → track real evals, not reward. Labelers → guidelines, agreement, calibration (and acknowledge whose values). Eval → human judgment, not the proxy. RLHF aligns a model to a measurement of human preference, so the measurement's quality and the optimization's honesty are everything.

You did it

You just designed an RLHF pipeline.

• A —
• R — L
• S — F
• P — r
• A —
• P — P
• D — P