InstructGPT,
explained.

Rung 1, so the comparison is against the pretrained model it starts from. A language model is trained to predict likely continuations of text, and that is simply not the same task as doing what a user asked. Given an instruction, the most likely continuation in the training corpus might be more instructions, a restatement, or a plausible-sounding evasion. The model is not failing; it is succeeding at a different objective from the one you want. Naming that as an objective mismatch rather than a capability gap is this rung's framing contribution.

The fix is a three-step pipeline, and every rung above modifies exactly one of its steps. One: collect demonstrations of the desired behaviour and fine-tune on them supervised. Two: collect human rankings of several model outputs for the same prompt and train a reward model to predict them. Rankings rather than scores is a deliberate choice — people are consistent when comparing two things and inconsistent when assigning absolute numbers. Three: optimise the model against that reward with PPO, plus a penalty for drifting too far from the step-one model, because a reward model is only accurate near the outputs it was trained on.

The unit to carry up the ladder is the result that made this recipe universal: a 1.3B-parameter InstructGPT was preferred by human evaluators over the 175B base model it was derived from. Same pretraining family, roughly a hundredth of the parameters, better outputs. That is the strongest statement anywhere on this site that post-training is not polish — it is a capability multiplier large enough to substitute for two orders of magnitude of scale.

GPT-3 was trained on one objective: predict the next word across the internet. That makes it astonishingly capable and subtly misaligned. Its goal is "what text is likely to come next," which is not the same as "what would be helpful, honest, and safe to say." The two often coincide — and often don't.

So raw GPT-3 would answer a question with another question (a plausible continuation), invent citations, ignore an instruction it found less likely, or happily produce toxic text if the prompt leaned that way. The model wasn't broken; it was doing exactly what it was trained to do. The problem was that next-word prediction is a proxy for helpfulness, not the thing itself. InstructGPT's question: how do you train for the thing itself, when "helpful" has no loss function?

You can't write down a formula for "a good answer." But a person can look at two answers and say which is better. InstructGPT's insight is to turn that human judgment into a training signal — a technique called reinforcement learning from human feedback, or RLHF.

The plan has three stages: first teach the model the general shape of good answers from human demonstrations; then distill thousands of human comparisons into a reward model that can score any answer; then let the language model practice against that reward, improving toward what people prefer. Humans can't rank every output the model will ever produce — so you train a model to imitate their taste, and optimize against that.

Start simple. Collect a set of real prompts (many from the OpenAI API), and have human labelers write the ideal response to each. Then fine-tune GPT-3 on these prompt→response pairs with ordinary supervised learning. This is SFT: the model learns, by imitation, the general format and manner of a helpful answer — follow the instruction, stay on topic, be direct.

SFT alone already makes a big difference. But it has a ceiling: it can only imitate the specific demonstrations written, and writing demonstrations is slow and expensive. Worse, for any prompt there are many good answers and the model has no way to learn which is better — imitation treats every demonstration as equally correct. To go further, you need a signal about preference, not just a single target.

Ranking is cheaper than writing. So instead of asking labelers to author perfect answers, InstructGPT shows them several answers the SFT model generated for the same prompt and asks them to rank them best to worst. These comparisons are the raw material of alignment.

A separate reward model — itself a Transformer — is then trained to predict those human preferences: given a prompt and a candidate answer, output a single scalar "how much a human would like this." Trained on enough comparisons, it becomes a stand-in for human judgment that can score any answer instantly, at scale, without a person in the loop. It's how you compress "human taste" into something an optimizer can chase.

The objective that turns "A beats B" into a scalar reward is Bradley-Terry. Model the probability a human prefers answer A as a sigmoid of the reward gap, then train the reward model to maximize the probability of the human's actual choice:

That single formula is the heart of RLHF — and the same loss DPO later optimizes directly, skipping the RL.

It's simpler than it looks: push the winner's reward up and the loser's down until the gaps reproduce the rankings. Trained on enough pairs, the scalar it outputs is a model of human taste — and the runnable version below recovers a full ranking from comparisons alone, no scores ever shown.

Now the model can practice. Treat the language model as a policy: it takes a prompt, generates an answer, and the reward model scores it. Using PPO (Proximal Policy Optimization), the policy is nudged to produce answers the reward model rates highly. Over many rounds, the model drifts toward outputs humans prefer — more helpful, more on-instruction, less toxic.

There's a crucial safety belt. If you let the policy optimize the reward model freely, it will find degenerate outputs that game the score — reward hacking — and drift into gibberish the reward model happens to like. So a KL-divergence penalty tethers the policy to the original SFT model, punishing it for straying too far. The result is aligned, but still fluent and sane.

The headline finding is startling: human labelers preferred the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3 — a model over a hundred times smaller. Alignment beat scale. InstructGPT followed instructions more faithfully, made up facts less often, and produced toxic output less frequently, all while being far cheaper to run.

There was a cost the paper names honestly — an alignment tax: aligning the model slightly dented performance on some raw NLP benchmarks. But by mixing in a little pre-training gradient during RL, they kept the tax small. The trade — a touch of benchmark score for a lot of helpfulness — was plainly worth it.

InstructGPT is aligned — but aligned to whom? The preferences come from a small pool of contractors following OpenAI's guidelines. "Helpful and harmless" is defined by that specific group, and the paper is candid that this is not a neutral, universal standard. Scale the method and you scale one set of values.

It also doesn't cure hallucination — the model still states falsehoods confidently, just less often. A determined user can still jailbreak it into harmful output. And the reward model is a leaky proxy: optimize hard enough and the policy learns to please the reward model rather than actual humans. RLHF made models dramatically more useful without making them truthful or safe by construction — those remain open problems.

InstructGPT is the missing half of the modern assistant. GPT-3 supplied the raw capability; InstructGPT supplied the alignment that made that capability usable by ordinary people typing ordinary requests. A few months after this paper, the same recipe — SFT, reward model, PPO — wrapped in a chat interface, became ChatGPT.

RLHF is now the default finishing step for frontier models, and the ideas here seeded a whole research line: reward modeling, preference optimization, and lighter-weight successors like DPO that skip the separate reward model entirely. If GPT-3 proved a big model is a powerful engine, InstructGPT proved that human preference is the steering wheel — and that steering can matter more than horsepower.

Three admissions, all of which became research areas. The alignment tax: the aligned model gets worse on some academic benchmarks than the base model, and the paper reports this rather than selecting metrics that hide it. Whose preferences: the model is aligned to the preferences of a small group of contractors and a handful of researchers' guidelines, which the paper states plainly — "aligned" is a relation to a specific group, not a property. And reward over-optimisation: push hard enough against a learned reward and you find its errors rather than the behaviour it was meant to encode, which is what the drift penalty exists to contain.

The pipeline's operational cost is also real and under-stated at the time: step three holds four models in play at once, and reinforcement learning on a language model is delicate to keep stable.

How it held up: the three steps became the industry default within a year, and the fact that a chat assistant feels like a different kind of artefact from a language model is this paper's doing. Every rung above is an attack on one of its three steps — rungs 2 and 3 on where the data comes from, rungs 4 and 5 on the cost of the gradient step, rungs 6 and 7 on the optimiser, rungs 8 and 9 on what the reward measures. The admission that has aged most importantly is the second one, because it reframes alignment as a question about whose values are being encoded — which is what the next rung makes explicit and auditable.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →