Constitutional AI,
explained.

InstructGPT proved the recipe: collect human preferences, train a reward model, optimize with RL. But scale the harmlessness half of that and the costs pile up. You need humans to read — and rank — the model's most disturbing outputs, by the tens of thousands. The labels arrive slow, expensive, and inconsistent, because reasonable people disagree about edge cases.

Worse, the resulting values are implicit. Ask "what does this model consider harmful, and why?" and the honest answer is: whatever pattern the reward model absorbed from a mountain of individual judgments nobody can inspect. If you want to change the values, you relabel the mountain.

The paper's core move is administrative in the best way: replace the mountain of judgments with a constitution — a short list of natural-language principles like "choose the response that is least likely to assist a dangerous act" or "choose the response a wise, peaceful person would give." Principles drawn from sources as varied as the UN Declaration of Human Rights and platform terms of service.

The constitution does nothing by itself. Its power comes from the two training phases that apply it — one supervised, one reinforcement.

Start with a helpful-only model — capable, compliant, and happy to answer harmful prompts. Feed it exactly those prompts. Then, in the same context, ask it to critique its own response against a randomly drawn principle — and then to revise the response to satisfy the critique.

The revisions — not the drafts — become the supervised fine-tuning data. The striking part: the same model that produced the harmful draft is perfectly capable of identifying why it's harmful and writing a better one, when asked. The knowledge was in there; the constitution gives it a job.

Phase two swaps the human out of the RLHF preference loop. Sample two responses to each harmful prompt, then ask a model: "According to this principle, which response is better?" Those AI judgments — millions of them, generated at machine speed — train the preference model, and the assistant is optimized against it with RL, exactly as in RLHF.

Human feedback still trains helpfulness — humans remain the authority on what's useful. But harmlessness, the part that burned labelers and budgets, now scales with compute instead of crowds. This is RLAIF, and it outperformed human-feedback harmlessness training in the paper's own evaluations.

Earlier safety training had an ugly side effect: models learned that the safest answer is no answer. "I can't help with that" — even for questions that deserved engagement. Evasion scores as harmless, but it's useless, and users hate it.

The critique-revision loop fixes this structurally: a stonewall response critiqued against "be helpful where you safely can" gets revised into an answer that engages, explains, and declines only the genuinely harmful part. The paper's models were rated both more harmless and less evasive than their RLHF baselines — breaking a trade-off everyone had assumed was fundamental.

Constitutional AI moves the alignment question; it doesn't dissolve it. Someone still writes the constitution — the values are explicit now, but they're still chosen, and contested choices don't become uncontested by being written down. The AI judge inherits the biases of the model doing the judging. And principles are interpreted by the model, sometimes in ways their authors wouldn't endorse — a constitution constrains behavior; it doesn't guarantee comprehension.

There's also a subtler dependency: the whole method leans on the base model already being capable enough to critique, revise and judge well. CAI harvests alignment from capability — which is exactly why it scaled beautifully as models got smarter, and why it says little about aligning systems smarter than their judges. That open end became its own research field: scalable oversight.

Constitutional AI is the backbone of how Claude was aligned, and RLAIF became a standard tool across the industry — AI feedback now trains reward models everywhere, for far more than safety. The deeper legacy is the reframing: alignment data is something a capable model can help generate, and values are something you can version-control.

Its intellectual descendants are everywhere: model-written critiques in eval pipelines, LLM-as-judge, self-improvement loops, and every "the model checks the model" pattern in production. When your guardrails pipeline asks one model to grade another against a written policy, you're running a small constitution.