InstructGPT,
explained.

Rung 1, so the comparison is against the pretrained model it starts from. A language model is trained to predict likely continuations of text, and that is simply not the same task as doing what a user asked. Given an instruction, the most likely continuation in the training corpus might be more instructions, a restatement, or a plausible-sounding evasion. The model is not failing; it is succeeding at a different objective from the one you want. Naming that as an objective mismatch rather than a capability gap is this rung's framing contribution.

The fix is a three-step pipeline, and every rung above modifies exactly one of its steps. One: collect demonstrations of the desired behaviour and fine-tune on them supervised. Two: collect human rankings of several model outputs for the same prompt and train a reward model to predict them. Rankings rather than scores is a deliberate choice — people are consistent when comparing two things and inconsistent when assigning absolute numbers. Three: optimise the model against that reward with PPO, plus a penalty for drifting too far from the step-one model, because a reward model is only accurate near the outputs it was trained on.

The unit to carry up the ladder is the result that made this recipe universal: a 1.3B-parameter InstructGPT was preferred by human evaluators over the 175B GPT-3 model. Same pretraining family, roughly a hundredth of the parameters, better outputs. That is the strongest statement anywhere on this site that post-training is not polish — it is a capability multiplier large enough to substitute for two orders of magnitude of scale.

GPT-3 was trained on one objective: predict the next word across the internet. That makes it astonishingly capable and subtly misaligned. Its goal is "what text is likely to come next," which is not the same as "what would be helpful, honest, and safe to say." The two often coincide — and often don't.

So raw GPT-3 would answer a question with another question (a plausible continuation), invent citations, ignore an instruction it found less likely, or happily produce toxic text if the prompt leaned that way. The model wasn't broken; it was doing exactly what it was trained to do. The problem was that next-word prediction is a proxy for helpfulness, not the thing itself. InstructGPT's question: how do you train for the thing itself, when "helpful" has no loss function?

You can't write down a formula for "a good answer." But a person can look at two answers and say which is better. InstructGPT's insight is to turn that human judgment into a training signal — a technique called reinforcement learning from human feedback, or RLHF.

The plan has three stages: first teach the model the general shape of good answers from human demonstrations; then distill thousands of human comparisons into a reward model that can score any answer; then let the language model practice against that reward, improving toward what people prefer. Humans can't rank every output the model will ever produce — so you train a model to imitate their taste, and optimize against that.

Start simple. Collect a set of real prompts (many from the OpenAI API), and have human labelers write the ideal response to each. Then fine-tune GPT-3 on these prompt→response pairs with ordinary supervised learning. This is SFT: the model learns, by imitation, the general format and manner of a helpful answer — follow the instruction, stay on topic, be direct.

SFT alone already makes a big difference. But it has a ceiling: it can only imitate the specific demonstrations written, and writing demonstrations is slow and expensive. Worse, for any prompt there are many good answers and the model has no way to learn which is better — imitation treats every demonstration as equally correct. To go further, you need a signal about preference, not just a single target.

Ranking is cheaper than writing. So instead of asking labelers to author perfect answers, InstructGPT shows them several answers the SFT model generated for the same prompt and asks them to rank them best to worst. These comparisons are the raw material of alignment.

A separate reward model — itself a Transformer — is then trained to predict those human preferences: given a prompt and a candidate answer, output a single scalar "how much a human would like this." Trained on enough comparisons, it becomes a stand-in for human judgment that can score any answer instantly, at scale, without a person in the loop. It's how you compress "human taste" into something an optimizer can chase.

The objective that turns "A beats B" into a scalar reward is Bradley-Terry. Model the probability a human prefers answer A as a sigmoid of the reward gap, then train the reward model to maximize the probability of the human's actual choice:

That single formula is the heart of RLHF — and the same loss DPO later optimizes directly, skipping the RL.

It's simpler than it looks: push the winner's reward up and the loser's down until the gaps reproduce the rankings. Trained on enough pairs, the scalar it outputs is a model of human taste — and the runnable version below recovers a full ranking from comparisons alone, no scores ever shown.

Now the model can practice. Treat the language model as a policy: it takes a prompt, generates an answer, and the reward model scores it. Using PPO (Proximal Policy Optimization), the policy is nudged to produce answers the reward model rates highly. Over many rounds, the model drifts toward outputs humans prefer — more helpful, more on-instruction, less toxic.

There's a crucial safety belt. If you let the policy optimize the reward model freely, it will find degenerate outputs that game the score — reward hacking — and drift into gibberish the reward model happens to like. So a KL-divergence penalty tethers the policy to the original SFT model, punishing it for straying too far. The result is aligned, but still fluent and sane.

InstructGPT is aligned — but aligned to whom? The preferences come from a small pool of contractors following OpenAI's guidelines. "Helpful and harmless" is defined by that specific group, and the paper is candid that this is not a neutral, universal standard. Scale the method and you scale one set of values.

It also doesn't cure hallucination — the model still states falsehoods confidently, just less often. A determined user can still jailbreak it into harmful output. And the reward model is a leaky proxy: optimize hard enough and the policy learns to please the reward model rather than actual humans. RLHF made models dramatically more useful without making them truthful or safe by construction — those remain open problems.

InstructGPT is the missing half of the modern assistant. GPT-3 supplied the raw capability; InstructGPT supplied the alignment that made that capability usable by ordinary people typing ordinary requests. A few months after this paper, the same recipe — SFT, reward model, PPO — wrapped in a chat interface, became ChatGPT.

RLHF is now the default finishing step for frontier models, and the ideas here seeded a whole research line: reward modeling, preference optimization, and lighter-weight successors like DPO that skip the separate reward model entirely. If GPT-3 proved a big model is a powerful engine, InstructGPT proved that human preference is the steering wheel — and that steering can matter more than horsepower.

InstructGPT starts from GPT-3 checkpoints at three sizes, 1.3B, 6B and 175B parameters, and from real prompts that early customers sent to the OpenAI API. About 40 contractors, hired through Upwork and Scale AI and screened for how well they spotted potentially harmful outputs, wrote the demonstrations and the rankings. Everything downstream inherits their judgment, and the paper reports how often they agreed with each other: about 73% of the time.

1. Build the prompt set. API prompts were deduplicated, filtered for personal information and split by user ID, so no customer appears in both training and evaluation. Because the API at first received few instruction-style prompts, labelers also wrote three kinds of their own: plain tasks, few-shot prompts with several query and response pairs, and prompts based on use cases described in API waitlist applications.
2. Supervised fine-tuning on demonstrations. Labelers wrote the desired output for about 13k training prompts. GPT-3 was fine-tuned on them for 16 epochs with cosine learning-rate decay and residual dropout of 0.2. Validation loss began to overfit after one epoch, but more epochs kept improving reward-model scores and human ratings, so the checkpoint was chosen by reward-model score rather than by loss.
3. Collect rankings, not scores. For about 33k prompts, labelers ranked between K = 4 and K = 9 model outputs from best to worst. One ranking implies K(K − 1)/2 pairwise comparisons, from 6 to 36 per prompt.
4. Train a 6B reward model. The reward model is a 6B SFT model whose unembedding layer is replaced by a head that outputs one scalar. All comparisons from one prompt form a single batch element: each output needs one forward pass instead of one per pair, and shuffling the correlated pairs into a flat dataset made the model overfit within one pass. A 175B reward model trained unstably, so every PPO run used the 6B one. Because the loss only sees reward differences, a bias was added so labeler demonstrations average a reward of 0 before RL.
5. Optimize with PPO in a bandit setting. An episode is one prompt from about 31k API prompts and one sampled response; the reward model scores the full response and the episode ends. A per-token KL penalty against the SFT model is subtracted, the value function is initialized from the reward model, and training ran for 256k episodes in batches of 512 with a PPO clip ratio of 0.2 and a KL coefficient of β = 0.02.
6. Mix in pretraining gradients (PPO-ptx). Plain PPO made the model worse on several public NLP benchmarks. PPO-ptx adds a language-modeling term on pretraining text, eight pretraining examples per RL episode, weighted by γ = 27.8. The models the paper calls InstructGPT are the PPO-ptx ones.
7. Evaluate on held-out customers. Labelers compared outputs on prompts from customers absent from training, rated overall quality on a 1–7 scale and flagged specific failures such as made-up facts or ignored constraints. A separate group of held-out labelers, who wrote no training data, repeated the comparisons.

Four networks in training, one at inference

The PPO step keeps four networks busy: the policy and the value network, both being trained, plus a frozen SFT copy for the KL term and the frozen reward model. Only the policy ships. At inference InstructGPT is an ordinary language model that samples a response; the reward model and the PPO machinery leave nothing behind except the weights they shaped.

The reward model is trained on every pair from a ranking at once:

The policy then maximizes:

• C(K, 2) — the number of pairs in one ranking; dividing by it gives every prompt equal weight however many outputs were ranked.
• rθ(x, y) — the 6B reward model's scalar for prompt x and response y; with the logistic σ, each pair is a Bradley–Terry comparison.
• β · log(π^RL / π^SFT) — the KL penalty, applied per token; β = 0.02.
• γ · E[log π^RL(x)] — ordinary language-modeling likelihood on pretraining text; γ = 27.8. With γ = 0 the paper calls the model PPO, with γ > 0 PPO-ptx.

At the extremes: β → 0 lets the policy chase the reward model's mistakes, while a very large β pins it to the SFT model and nothing is learned. A very large γ turns training back into language modeling. The chosen γ greatly reduced the benchmark regressions without lowering labeler preference scores.

The main metric was how often labelers preferred a model's output over the 175B SFT model's, on prompts from customers held out of training. Baselines were GPT-3, GPT-3 with a carefully written few-shot prompt, and 175B GPT-3 fine-tuned on the public FLAN and T0 instruction collections. Public benchmarks covered truthfulness, toxicity, bias and standard NLP tasks.

On TruthfulQA the introduction reports truthful and informative answers about twice as often as GPT-3; the results discussion calls the gain small but significant, and the 1.3B PPO-ptx model was slightly worse than GPT-3 of its size. Held-out labelers preferred InstructGPT at about the same rate as the labelers who produced training data, and reward models trained on some labeler groups predicted a held-out group's preferences only slightly less accurately.

The negative results matter as much. Plain PPO regressed on SQuAD, DROP, HellaSwag and WMT 2015 French-to-English translation; PPO-ptx greatly reduced that alignment tax and beat GPT-3 on HellaSwag, but still trailed on DROP, SQuADv2 and translation. Without the respectful prompt the toxicity advantage disappeared, and when explicitly asked for toxic text, InstructGPT produced much more of it than GPT-3. It had learned to follow instructions, including bad ones.

Three admissions, all of which became research areas. The alignment tax: the aligned model gets worse on some academic benchmarks than the base model, and the paper reports this rather than selecting metrics that hide it. Whose preferences: the model is aligned to the preferences of a small group of contractors and a handful of researchers' guidelines, which the paper states plainly — "aligned" is a relation to a specific group, not a property. And reward over-optimisation: push hard enough against a learned reward and you find its errors rather than the behaviour it was meant to encode, which is what the drift penalty exists to contain.

The pipeline's operational cost is also real and under-stated at the time: step three holds four models in play at once, and reinforcement learning on a language model is delicate to keep stable.

How it held up: the three steps became the industry default within a year, and the fact that a chat assistant feels like a different kind of artefact from a language model is this paper's doing. Every rung above is an attack on one of its three steps — rungs 2 and 3 on where the data comes from, rungs 4 and 5 on the cost of the gradient step, rungs 6 and 7 on the optimiser, rungs 8 and 9 on what the reward measures. The admission that has aged most importantly is the second one, because it reframes alignment as a question about whose values are being encoded — which is what the next rung makes explicit and auditable.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →