Constitutional AI,
explained.

Rung 1's harmlessness comes from humans labelling harmful outputs, which has two problems that have nothing to do with each other. It is expensive and unpleasant — the labelling involves people reading the worst outputs the model can produce — and the resulting values are implicit, distributed across thousands of individual judgements with no artefact anyone can read, disagree with, or audit.

The change addresses both with one move: state the principles explicitly as a constitution, and have the model apply them to itself. First a supervised phase, in which the model produces a response, critiques it against a principle, and revises it, with the revisions becoming training data. Then a reinforcement phase in which the preference comparisons that rung 1 got from humans are generated by a model choosing which of two responses better satisfies the constitution — reinforcement learning from AI feedback rather than human feedback.

Stated in rung 1's units: the pipeline is the same three steps, with human harm labels reduced to near zero while human input moves to writing and revising the principles themselves. There is also a second claim that is about quality rather than cost, and it is the more interesting one. Models trained to be harmless by preference labelling tend toward evasion — refusing flatly, declining to engage, treating the safest output as the shortest one. A model that has been trained to critique itself against a stated reason has the reason available, so it can decline and explain why, which is a different and considerably more useful behaviour.

InstructGPT proved the recipe: collect human preferences, train a reward model, optimize with RL. But scale the harmlessness half of that and the costs pile up. You need humans to read — and rank — the model's most disturbing outputs, by the tens of thousands. The labels arrive slow, expensive, and inconsistent, because reasonable people disagree about edge cases.

Worse, the resulting values are implicit. Ask "what does this model consider harmful, and why?" and the honest answer is: whatever pattern the reward model absorbed from a mountain of individual judgments nobody can inspect. If you want to change the values, you relabel the mountain.

The paper's core move is administrative in the best way: replace the mountain of judgments with a constitution — a short list of natural-language principles like "choose the response that is least likely to assist a dangerous act" or "choose the response a wise, peaceful person would give." In the paper the principles were a short list the authors wrote themselves; it was Anthropic's later public constitution for Claude (2023) that drew on sources such as the UN Declaration of Human Rights.

The constitution does nothing by itself. Its power comes from the two training phases that apply it — one supervised, one reinforcement.

Start with a helpful-only model — capable, compliant, and happy to answer harmful prompts. Feed it exactly those prompts. Then, in the same context, ask it to critique its own response against a randomly drawn principle — and then to revise the response to satisfy the critique.

The revisions — not the drafts — become the supervised fine-tuning data. The striking part: the same model that produced the harmful draft is perfectly capable of identifying why it's harmful and writing a better one, when asked. The knowledge was in there; the constitution gives it a job.

Phase two swaps the human out of the RLHF preference loop. Sample two responses to each harmful prompt, then ask a model: "According to this principle, which response is better?" Those AI judgments — on the order of 180,000 of them in the paper, generated at machine speed — train the preference model, and the assistant is optimized against it with RL, exactly as in RLHF.

Human feedback still trains helpfulness — humans remain the authority on what's useful. But harmlessness, the part that burned labelers and budgets, now scales with compute instead of crowds. This is RLAIF, and it outperformed human-feedback harmlessness training in the paper's own evaluations.

Both phases are the same three operations against the written constitution, and no human ever labels a harm:

The model finds its own violations, writes the fix, and labels which of two answers is better — a full training signal with the constitution as the only authority. The runnable version below does exactly this.

Earlier safety training had an ugly side effect: models learned that the safest answer is no answer. "I can't help with that" — even for questions that deserved engagement. Evasion scores as harmless, but it's useless, and users hate it.

The critique-revision loop fixes this structurally: a stonewall response critiqued against "be helpful where you safely can" gets revised into an answer that engages, explains, and declines only the genuinely harmful part. The paper's models were rated both more harmless and less evasive than their RLHF baselines — breaking a trade-off everyone had assumed was fundamental.

Constitutional AI moves the alignment question; it doesn't dissolve it. Someone still writes the constitution — the values are explicit now, but they're still chosen, and contested choices don't become uncontested by being written down. The AI judge inherits the biases of the model doing the judging. And principles are interpreted by the model, sometimes in ways their authors wouldn't endorse — a constitution constrains behavior; it doesn't guarantee comprehension.

There's also a subtler dependency: the whole method leans on the base model already being capable enough to critique, revise and judge well. CAI harvests alignment from capability — which is exactly why it scaled beautifully as models got smarter, and why it says little about aligning systems smarter than their judges. That open end became its own research field: scalable oversight.

Constitutional AI is the backbone of how Claude was aligned, and RLAIF became a standard tool across the industry — AI feedback now trains reward models everywhere, for far more than safety. The deeper legacy is the reframing: alignment data is something a capable model can help generate, and values are something you can version-control.

Its intellectual descendants are everywhere: model-written critiques in eval pipelines, LLM-as-judge, self-improvement loops, and every "the model checks the model" pattern in production. When your guardrails pipeline asks one model to grade another against a written policy, you're running a small constitution.

The method needs four ingredients: a helpful-only RLHF assistant trained on human helpfulness preferences alone, a pretrained language model, a large prompt set and a short constitution. The main runs used 52-billion-parameter models. The supervised phase used 182,831 red-team prompts, 42,496 written by people in earlier red-teaming work and 140,335 generated by prompting a model, plus 135,296 human-written helpfulness prompts. The constitution was 16 principles the authors wrote for the research and describe as chosen in a fairly ad hoc way. Sourcing principles from documents such as the UN's Universal Declaration of Human Rights came later, in the constitution Anthropic published for Claude in May 2023.

1. Draft. The helpful-only model answers each red-team prompt. Trained only to be helpful, it often complies with the harmful request.
2. Critique. A critique request built from one randomly drawn principle is appended to the conversation, and the same model critiques its own draft. A few examples of critiques and revisions are prepended so the model knows the format.
3. Revise. The matching revision request asks the model to rewrite the draft to address the critique. This runs four times per prompt, with a freshly sampled principle each round.
4. Supervised fine-tuning (SL-CAI). A pretrained model, not the helpful RLHF model, is fine-tuned on the revisions from all four rounds, together with two responses per helpfulness prompt sampled from the helpful RLHF model, so harmlessness training does not erase helpfulness.
5. Sample comparison pairs. The SL-CAI model generates two responses to each red-team prompt.
6. Ask the feedback model. A pretrained model reads the conversation, a randomly drawn principle and the two responses presented as options (A) and (B), and is asked which better satisfies the principle. Its normalized probabilities for (A) and (B) become a soft preference label rather than a hard vote. Sampling a different principle for each label acts as an ensemble, and the authors found it more robust than relying on one principle.
7. Optionally, reason first. In the chain-of-thought variant, the helpful RLHF model is prompted to think step by step before choosing. Its probabilities came out near 0 or 1, so they were clamped to the 40–60% range, which worked better than a looser 20–80% clamp.
8. Train the preference model. Its data mixes 135,296 human helpfulness comparisons with 182,831 constitution-generated harmlessness comparisons. No person labels a single harmlessness comparison.
9. Reinforcement learning (RL-CAI). Starting from the SL-CAI model, PPO optimizes the policy against the preference model, adding 491,142 model-generated red-team prompts and 474,300 model-generated helpfulness prompts.

Why the labels are probabilities

A hard (A)-or-(B) label throws away how sure the feedback model was. Training on its normalized probabilities lets a near-tie count as a near-tie, so a borderline pair nudges the preference model gently instead of arriving as a confident vote. The chain-of-thought clamp applies the same logic from the other side: reasoning made the feedback model overconfident, and capping its labels at 60% stops that overconfidence passing into the preference model.

At inference the result is an ordinary assistant. Critique, revision and judging all happen during training; the deployed model runs no extra loop when it answers.

For a red-team prompt x, a sampled principle c and two responses, the feedback model's log-probabilities of answering (A) and (B), written ℓ_A and ℓ_B, are normalized into a soft label:

The preference model scores the two responses r_A and r_B and is fit to p with the usual pairwise loss, written here with a soft target:

• ℓA, ℓB — how likely the feedback model is to answer with each option after reading the principle.
• p — the soft label; 0.5 means the feedback model cannot tell the responses apart.
• σ(rA − rB) — the preference model's Bradley–Terry probability that A is better.

At the extremes, p = 1 reduces the loss to the hard-label loss used for human comparisons, and p = 0.5 pulls the two scores together. The clamp keeps chain-of-thought labels within 0.1 of a tie, so no single overconfident judgment can dominate.

All compared policies were 52B models. Crowdworkers held open-ended conversations in which each turn showed two responses from two different models and picked one; more than 18,000 such comparisons across 24 model snapshots were turned into helpfulness and harmlessness Elo scores, where only differences carry meaning. Unlike earlier work, crowdworkers were told to prefer a thoughtfully harmless response over an evasively harmless one. The baselines were a helpful-only RLHF model and an HH RLHF model trained on human helpfulness and human harmlessness labels.

The AI feedback was also tested directly. On 438 binary comparison questions, pretrained models picked the more helpful, honest and harmless response more accurately as they grew, and chain of thought helped further. The authors projected that models larger than 52B would be competitive with preference models trained on human feedback; they did not show parity at 52B.

Two ablations matter. Critiquing before revising produced more harmless revisions for smaller models, while for the 52B model direct revision did nearly as well, though critiques were always slightly better; they were kept because they make the reasoning inspectable. And RL-CAI could be over-trained: past a point the policy turned harsh or appended reassuring boilerplate to its answers to red-team prompts, a Goodhart effect in which the preference model's taste for caring language became a target in itself.

The central limitation is acknowledged directly and is worth stating carefully, because it is easy to caricature. Moving to AI feedback does not remove human values from the loop; it relocates them from thousands of unexamined judgements into a short document that someone chose to write. That is a genuine improvement in auditability and no improvement at all in objectivity — the constitution is a set of choices, and the paper presents it as one.

The method also has a capability precondition: the model must be good enough to evaluate its own output against an abstract principle. Where it is not, the self-critique produces noise, and the loop amplifies it. There is no way to bootstrap harmlessness into a model too weak to judge.

How it held up: both halves spread. AI feedback is now routine — the general pattern of using a capable model to generate preference or evaluation signal at a scale humans cannot match is standard practice across the industry, in evaluation as much as in training. And the norm of publishing an explicit specification of intended behaviour, so that a model's failures can be argued about against a stated intent rather than against an assumption, became a convention that several labs now follow. The open question the paper leaves is the one the field still has: a written principle is only as good as the disagreements it was written to survive.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

A pre-trained model completes text; it does not take instructions. Nine papers on getting from one to the other, from RLHF to a loss with no reward model in it.

1. 1InstructGPT2022
2. 2Constitutional AI2022
3. 3Self-Instruct2023
4. 4LoRA2022
5. 5QLoRA2023
6. 6Direct Preference Optimization2023
7. 7GRPO2024
8. 8STaR2022
9. 9Let's Verify Step by Step2023

All 110 papers, in 19 ladders →