InstructGPT,
explained.

GPT-3 was trained on one objective: predict the next word across the internet. That makes it astonishingly capable and subtly misaligned. Its goal is "what text is likely to come next," which is not the same as "what would be helpful, honest, and safe to say." The two often coincide — and often don't.

So raw GPT-3 would answer a question with another question (a plausible continuation), invent citations, ignore an instruction it found less likely, or happily produce toxic text if the prompt leaned that way. The model wasn't broken; it was doing exactly what it was trained to do. The problem was that next-word prediction is a proxy for helpfulness, not the thing itself. InstructGPT's question: how do you train for the thing itself, when "helpful" has no loss function?

You can't write down a formula for "a good answer." But a person can look at two answers and say which is better. InstructGPT's insight is to turn that human judgment into a training signal — a technique called reinforcement learning from human feedback, or RLHF.

The plan has three stages: first teach the model the general shape of good answers from human demonstrations; then distill thousands of human comparisons into a reward model that can score any answer; then let the language model practice against that reward, improving toward what people prefer. Humans can't rank every output the model will ever produce — so you train a model to imitate their taste, and optimize against that.

Start simple. Collect a set of real prompts (many from the OpenAI API), and have human labelers write the ideal response to each. Then fine-tune GPT-3 on these prompt→response pairs with ordinary supervised learning. This is SFT: the model learns, by imitation, the general format and manner of a helpful answer — follow the instruction, stay on topic, be direct.

SFT alone already makes a big difference. But it has a ceiling: it can only imitate the specific demonstrations written, and writing demonstrations is slow and expensive. Worse, for any prompt there are many good answers and the model has no way to learn which is better — imitation treats every demonstration as equally correct. To go further, you need a signal about preference, not just a single target.

Ranking is cheaper than writing. So instead of asking labelers to author perfect answers, InstructGPT shows them several answers the SFT model generated for the same prompt and asks them to rank them best to worst. These comparisons are the raw material of alignment.

A separate reward model — itself a Transformer — is then trained to predict those human preferences: given a prompt and a candidate answer, output a single scalar "how much a human would like this." Trained on enough comparisons, it becomes a stand-in for human judgment that can score any answer instantly, at scale, without a person in the loop. It's how you compress "human taste" into something an optimizer can chase.

Now the model can practice. Treat the language model as a policy: it takes a prompt, generates an answer, and the reward model scores it. Using PPO (Proximal Policy Optimization), the policy is nudged to produce answers the reward model rates highly. Over many rounds, the model drifts toward outputs humans prefer — more helpful, more on-instruction, less toxic.

There's a crucial safety belt. If you let the policy optimize the reward model freely, it will find degenerate outputs that game the score — reward hacking — and drift into gibberish the reward model happens to like. So a KL-divergence penalty tethers the policy to the original SFT model, punishing it for straying too far. The result is aligned, but still fluent and sane.

The headline finding is startling: human labelers preferred the 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3 — a model over a hundred times smaller. Alignment beat scale. InstructGPT followed instructions more faithfully, made up facts less often, and produced toxic output less frequently, all while being far cheaper to run.

There was a cost the paper names honestly — an alignment tax: aligning the model slightly dented performance on some raw NLP benchmarks. But by mixing in a little pre-training gradient during RL, they kept the tax small. The trade — a touch of benchmark score for a lot of helpfulness — was plainly worth it.

InstructGPT is aligned — but aligned to whom? The preferences come from a small pool of contractors following OpenAI's guidelines. "Helpful and harmless" is defined by that specific group, and the paper is candid that this is not a neutral, universal standard. Scale the method and you scale one set of values.

It also doesn't cure hallucination — the model still states falsehoods confidently, just less often. A determined user can still jailbreak it into harmful output. And the reward model is a leaky proxy: optimize hard enough and the policy learns to please the reward model rather than actual humans. RLHF made models dramatically more useful without making them truthful or safe by construction — those remain open problems.

InstructGPT is the missing half of the modern assistant. GPT-3 supplied the raw capability; InstructGPT supplied the alignment that made that capability usable by ordinary people typing ordinary requests. A few months after this paper, the same recipe — SFT, reward model, PPO — wrapped in a chat interface, became ChatGPT.

RLHF is now the default finishing step for frontier models, and the ideas here seeded a whole research line: reward modeling, preference optimization, and lighter-weight successors like DPO that skip the separate reward model entirely. If GPT-3 proved a big model is a powerful engine, InstructGPT proved that human preference is the steering wheel — and that steering can matter more than horsepower.