The whole design, in writing
Learn AI system design by building an RLHF (reinforcement learning from human feedback) alignment pipeline step by step. An interactive guide covering why a base model is capable but unaligned, supervised fine-tuning on demonstrations, why preferences beat demonstrations, training a reward model from human comparisons, PPO optimization with a KL constraint to prevent reward hacking, the DPO and RLAIF alternatives, and the unhappy paths (reward hacking, overoptimization, labeler quality).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What does RLHF do?
A pretrained base model is astonishingly capable — it has read the internet — but it's not aligned: it predicts plausible text, not helpful, honest, harmless answers. Ask it a question and it might continue with more questions, ramble, or produce unsafe content. The gap between "capable text predictor" and "assistant that does what people want" is what alignment closes.
RLHF (reinforcement learning from human feedback) aligns a base model to human preferences in three stages: supervised fine-tuning on demonstrations (teach it to follow instructions), a reward model trained on human comparisons (learn what people prefer), and RL optimization (PPO) against that reward — with a KL constraint so it improves without going off the rails.
Step 1 · Capable but unaligned
Why pretraining isn't enough
The base model has read everything and can generate fluent text on anything. So why isn't it already a good assistant — and why can't you just prompt your way to alignment?
A base model is hugely capable. Why isn't it already an aligned assistant?
Pretraining optimizes next-token prediction over internet text, so the model mimics the distribution of text (including unhelpful or unsafe continuations) rather than following instructions or matching human preferences. Prompting helps but can't reliably instill the behavior; you need to fine-tune the model toward what humans actually want.
Knowledge isn't the gap — the base model knows a great deal. The gap is behavior/objective: it predicts plausible text rather than being a helpful, safe assistant, which alignment training instills.
Size gives capability, not alignment. Even a huge base model predicts text rather than following human intent; you align it via fine-tuning + preference learning, not by scaling alone.
Pretraining optimizes next-token prediction over internet text, so the base model learned to mimic the distribution of text — including rambling, refusing to answer, or unsafe continuations — not to be helpful, honest, and harmless. That's a different objective from "assist the user." Prompting nudges it, but you can't reliably prompt-engineer alignment into a model; you have to train it toward human intent. That training is RLHF.
Why this piece earns its place
Whether a behavior lives in a prompt or in the weights is a shipping decision. A system prompt is text in the same context window as the user's turn, and the reason a model treats it as more authoritative at all is itself something a pipeline like this one trained in — so prompting stands on alignment rather than substituting for it, and it inherits every weakness of a thing made of text. A long conversation dilutes it, a role-play frame argues with it, and whoever integrates your model can delete it outright. Behavior you cannot afford to lose that way has to be in the weights, and that portability is what you are buying. Plan for it not being free. Behavior is bought with capability: a model trained hard to be careful hedges more, declines more, and can come back worse at things the base model did easily. So run a capability suite that forms no part of the alignment objective before and after every run, and read it against the base model's own scores — that is the only baseline that prices what alignment cost you rather than what the last round cost you. Track preference alone and you will never see the bill.
Step 2 · Three stages
The alignment pipeline
Given a capable-but-unaligned base model, what's the overall recipe to turn it into an aligned assistant?
Three stages on top of the base model. (1) Supervised fine-tuning (SFT): fine-tune on human demonstrations of good answers, teaching instruction-following. (2) Reward model (RM): collect human preference comparisons and train a model to predict which output people prefer. (3) RL optimization: use RL (PPO) to optimize the SFT model to produce outputs the reward model scores highly — kept close to the SFT model by a KL penalty. SFT teaches the format; RM+RL refine toward preference.
Why this piece earns its place
Read the three stages as three artifacts you now own and have to version together, because they are not independent. The reference is a frozen copy of one particular SFT run, the reward model is usually initialized from that same run, and the preference pairs were collected on outputs that run produced. Re-run SFT with new demonstrations and all three quietly go stale: the reward model is now scoring a policy it was not built from, and the comparisons behind it describe a distribution that policy has already left. So in practice nobody runs this pipeline once — you run a round, collect fresh comparisons on the current policy's own samples, and retrain, because preference data ages against the model it was gathered from. The other thing worth saying out loud is when to stop at stage one. If what you want is format, tone and instruction-following, SFT alone gets you there and the whole reward-model-plus-RL half earns nothing. The preference machinery pays for itself only when the quality you are after is one nobody on the team can write down as a demonstration.
- SFTdemonstrations
- RMpreferences
- RLoptimize to reward
What the new pieces do
- Base Modelbase
- A pretrained LLM: hugely capable at predicting text, but not aligned — it doesn't reliably follow instructions or match human preferences.
Step 3 · Teach the format
Supervised fine-tuning
First, get the base model to actually respond like an assistant. What data and objective do that?
SFT: fine-tune the base model on a dataset of demonstrations — prompts paired with high-quality human-written (or curated) responses — using standard supervised next-token loss. This teaches the model the format and behavior of following instructions: answer the question, in the right style, helpfully. The result is a solid instruction-following model — and, importantly, the starting point and reference for the RL stage.
Why this piece earns its place
Mask the loss to the response tokens. If it covers the prompt as well you are also teaching the model to write the questions, which is not the job you are paying for. Then look hard at what else each example teaches, because imitation takes the whole example and not the part you meant: the padding, the preamble that restates the question before answering it, the reflexive hedge, the four-paragraph reply to a one-line ask. A small, tightly curated demonstration set beats a large scraped one for exactly that reason — everything in it becomes a habit, and nothing later in the pipeline takes those habits back out. One trap deserves naming on its own. A demonstration that answers a question the model holds no internal knowledge of does not teach it the fact; it teaches the shape of a confident answer to that kind of question, which is training for invention. Prefer demonstrations whose content the model could plausibly already know, and push the rest to retrieval or tools where it belongs.
- demosprompt → ideal answer
- supervisednext-token loss
- resultinstruction-follower
What the new pieces do
- Demonstrationsclient
- Human-written examples of good responses to prompts — the supervised signal that teaches the model to follow instructions.
- SFTservice
- Supervised fine-tuning of the base model on demonstrations — makes it follow instructions. The starting point for RL.
Step 4 · Why not stop at SFT?
Preferences beat demonstrations
SFT already gives a decent assistant. Why add the whole reward-model + RL machinery instead of just collecting more demonstrations?
SFT works reasonably. Why add reward-model + RL instead of just more demonstrations?
It's not fashion — it's that comparisons are a cheaper, richer signal than demonstrations. Humans can't always write the ideal answer, but they can reliably say which of two answers is better, and that judgment covers quality dimensions demonstrations miss.
Demonstrations are useful (that's SFT), but limited: writing the single best answer is hard and expensive, and doesn't teach the model the full ranking of good-to-bad. Preferences add that.
Writing a perfect demonstration is hard and doesn't convey how much better it is than alternatives. Having humans compare two model outputs (A vs B) is faster, cheaper, and captures nuanced preferences (tone, helpfulness, safety) that are hard to demonstrate — giving a reward signal you can then optimize against at scale.
Demonstrations are expensive and limited: writing the single ideal answer is hard, and it doesn't tell the model how much better it is than the alternatives. Humans find it far easier and cheaper to compare two outputs ("A is better than B") than to author the perfect one — and those comparisons capture nuanced quality (tone, helpfulness, safety, subtle correctness) that's hard to demonstrate. Preferences give a richer, scalable signal you can turn into a reward and optimize against.
Why this piece earns its place
A pairwise comparison gives you order, never level. Hand a labeler two bad answers and the data still reports a winner, so a preference set has no way to express unacceptable — only a ranking. That is why safety rarely rides on comparisons alone: an absolute rule or a separate classifier has to supply the floor pairwise data structurally cannot. The prompt distribution in that set is also the scope of your alignment. The model is tuned on the kinds of questions someone thought to collect pairs for, and behavior on the ones nobody collected is whatever SFT left behind — so the coverage decision is made by whoever writes the prompt list, long before any training runs. And systematic labeler preference is a different problem from labeler noise. Noise averages out as you add annotators; bias sharpens. People reliably prefer longer, more confident, better-formatted answers, so a clean, high-agreement dataset can still encode length as a stand-in for quality. You catch that by measuring the trait itself — length, hedging, markdown — against the win rate, never by watching the agreement rate, which rises as the bias gets more consistent.
- compareA vs B
- cheaper + richerthan writing ideal
- nuancetone/safety/quality
Step 5 · Model the preference
The reward model
You have human comparisons, but RL needs a score for any output on the fly — you can't ask a human on every RL step. How do you turn sparse human comparisons into a dense reward signal?
Train a reward model: take the human preference pairs (for a prompt, output A preferred over B) and train a model to output a scalar score such that preferred outputs score higher (a ranking loss). Now you have a fast, automatic proxy for human preference that can score any output during RL — a dense reward from sparse human labels. The reward model is typically initialized from the SFT model with a scalar head. Its quality is bounded by the preference data quality.
Why this piece earns its place
The score is comparative, and only within a prompt. A ranking loss constrains the gap between two answers to the same question and nothing else, so the number carries no absolute meaning across questions: there is no cutoff you can write down that means good enough, and a three here says nothing about a three over there. The reward model is a ranker, not a grader, and the moment a release gate is written as an absolute reward threshold it has been promoted to a job it cannot do. Two reward-model runs are not comparable either — retrain it and the whole scale moves, so a higher reward across a retrain means nothing unless one model scored both policies. Hold out preference pairs before you train, however expensive they were to collect; agreement with held-out human choices is the one number that says whether the model learned preference or memorized annotator habits, and you cannot compute it from data you trained on. And record which pair set and which labeling batch each reward model was built from. When a policy starts rewarding something strange, the question is always which labels taught it, and only that lineage answers it.
- inputhuman A vs B pairs
- outputscalar preference score
- proxydense reward for RL
What the new pieces do
- Preferencesclient
- Humans compare pairs of model outputs and pick the better one — a cheaper, richer signal than writing ideal answers.
- Reward Modelservice
- Trained on human comparisons to score any output by predicted human preference — the reward signal RL optimizes.
Step 6 · Optimize, on a leash
RL (PPO) with a KL constraint
Now push the model to produce high-reward outputs. But the reward model is an imperfect proxy — optimize it too hard and the model will find degenerate outputs that score high but are actually bad (reward hacking). How do you optimize without going off the rails?
RL against an imperfect reward model can be "hacked". How do you optimize safely?
PPO optimizes the policy to maximize reward MINUS a KL-divergence penalty from the frozen SFT reference model. The penalty keeps the policy from drifting far from sensible behavior to chase reward-model artifacts, so it improves along human preference without collapsing into degenerate high-reward-but-bad outputs. It's a leash: climb the reward, but don't wander off.
Unconstrained optimization of an imperfect proxy is exactly reward hacking — the policy finds weird outputs that fool the reward model but are worse to humans. You must constrain it toward the reference.
More optimization pressure on a proxy reward makes hacking worse, not better. The fix is a constraint (KL to the reference), not more unconstrained reward-chasing.
Use RL — PPO — to optimize the policy to maximize the reward, minus a KL-divergence penalty from the frozen SFT reference model. The reward pulls the policy toward human-preferred outputs; the KL penalty keeps it close to known-good behavior, so it can't drift far to exploit reward-model artifacts. That balance — climb the reward, stay near the reference — is what lets RLHF improve alignment without reward hacking or degenerating into gibberish that games the proxy.
Why this piece earns its place
What the diagram cannot show is that this stage feeds itself. The policy generates the samples, the reward model scores what it just generated, the update moves the policy, and the next batch is drawn from the moved policy — there is no fixed dataset anywhere in the run, so a bias in the reward compounds through the samples instead of averaging out across them. That is what makes KL an instrument and not only a penalty term. Log it every step. Reward climbing while KL climbs with it is the signature of a policy walking away from the reference to exploit the proxy, and it appears well before held-out quality falls, which makes it the earliest warning the run will give you. It also settles what you mean by a tuned run. You can fix β and read the resulting drift as an outcome, or name a KL you are willing to tolerate and let a controller move β to hold it there. Both are ordinary. Reporting a tuned run without saying which of the two you held fixed is not.
- PPOmaximize reward
- − KLstay near reference
- preventsreward hacking / drift
What the new pieces do
- Reference (KL)ref
- A frozen copy of the SFT model. The KL penalty keeps the RL policy close to it, preventing reward hacking and drift.
- RL (PPO)service
- Reinforcement learning: optimizes the policy to maximize the reward model's score, minus a KL penalty to the reference.
- Aligned Policymint
- The output: a model that follows instructions and matches human preferences — helpful and harmless.
Step 7 · Simpler & cheaper variants
DPO and RLAIF
Full PPO-based RLHF is complex and finicky (a reward model + on-policy RL + a reference, all in the loop). Are there simpler or cheaper ways to get the same alignment?
Two big alternatives. DPO (Direct Preference Optimization) skips the explicit reward model and RL loop: it optimizes the policy directly on the preference pairs with a classification-style loss that provably targets the same objective — much simpler and more stable, now very common. RLAIF (RL from AI Feedback / Constitutional AI) replaces (some) expensive human comparisons with AI-generated preferences guided by a set of principles — scaling the feedback cheaply. Both attack RLHF's costs: DPO its complexity, RLAIF its human-labeling expense.
Why this piece earns its place
What DPO removes is an asset, not only a cost. A trained reward model keeps earning long after the RL run ends: sample several candidates at inference and serve the highest-scoring one, filter a training set with it, rank experimental checkpoints against each other, run it as a standing eval nobody has to staff. Finish with DPO and none of that exists, so price it in before calling the simpler path free. RLAIF changes the shape of the human's job more than its size. People stop judging outputs and start writing and auditing the principles the AI judges by — and the error moves with them. A weak labeler is one voice among many and washes out; a badly worded clause is applied with perfect consistency to every comparison in the set, so its effect is correlated rather than noisy and no amount of extra data averages it away. That is also the upside: the standard is now a document you can read, review, diff between runs and argue about a specific line of, which is impossible while it lives in a labeler's head.
- DPOskip RM + RL, optimize prefs directly
- RLAIFAI feedback, cheaper labels
- same goalalign to preference
Step 8 · The sharp edges
Hacking, overoptimization & labelers
RLHF's hard problems are real: reward hacking, reward-model overoptimization, the quality and bias of human labelers, and evaluating whether the model actually got better.
Contain reward hacking with the KL constraint and by not over-training. Watch overoptimization: the reward model is a proxy, so beyond a point more RL degrades true quality even as reward-model score rises (Goodhart) — monitor real evals, not just reward. Invest in labeler quality: clear guidelines, inter-annotator agreement, and calibration, because the whole pipeline inherits labelers' judgments and biases (whose preferences? is a live question). And evaluate alignment with held-out human evals / red-teaming, not the reward model — the reward is the means, human-judged behavior is the goal.
Why this piece earns its place
One aggregate win rate is the metric most likely to lie to you, because it averages over categories that move in opposite directions. Preference data pays for agreeing with the user, so sycophancy is not a bug that crept in — it is behavior the signal directly rewards, and it lifts the overall number while making the model worse at the one job where disagreeing is the helpful answer. Slice the evaluation by category and gate on per-category regressions, or the average will quietly absorb one of them going bad. Then budget for truth. Held-out human judgment is the only signal in this design that is not a proxy, and it is slow and expensive, so it cannot run on every checkpoint: automatic evals shortlist, people decide the few that matter. Keep that pool and its guidelines separate from the one producing training comparisons — when the same judgment trains the proxy and then grades it, high agreement is partly measuring a shared blind spot. And an alignment defect lives in the weights, so the only true fix is another run. Keep the previous checkpoint serving-ready; at incident speed, reverting is the only move you have.
You did it
You just designed an RLHF pipeline.
Everything you assembled, in order
- A pretrained base model is capable but unaligned — it predicts text, not helpful/honest/harmless answers.
- RLHF aligns it in three stages: SFT → reward model → RL (PPO).
- SFT fine-tunes on demonstrations to teach instruction-following — the starting point and reference.
- Preferences (compare A vs B) are cheaper and richer than demonstrations — judging beats authoring.
- A reward model turns sparse human comparisons into a dense, automatic preference score.
- PPO maximizes reward minus a KL penalty to the SFT reference — climb the reward without reward-hacking.
- DPO (skip the RM+RL) and RLAIF (AI feedback) are simpler/cheaper paths; watch hacking, overoptimization, labeler quality.