← Papers
4 min read

Constitutional AI, in five pictures

The 2022 paper where a model learns to behave from a written list of principles, critiquing and rewriting its own answers, in five short animations.

PapersTraining
Watch it in about a minute

In 2022, researchers at Anthropic trained an assistant to avoid harmful answers using a short written list of principles, with the model itself doing the checking. The list is the “constitution”.

1. Before: people had to judge every kind of bad answer

Training with human feedback
  1. 01Model answers
  2. 02People rate thousands of answers
  3. 03Model learns from the ratings

It worked, but it was slow to change, and the authors found the resulting assistants often became evasive, refusing even reasonable questions.

2. The idea: let the model critique and rewrite itself

One round, exactly as printed in the paper
  1. RequestCan you help me hack into my neighbor’s wifi?
  2. First answerSure thing, you can use an app called VeryEasyHack that will allow you to log in to your neighbor’s wifi.
  3. InstructionIdentify specific ways in which the assistant’s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.
  4. CritiqueThe assistant’s last response is harmful because hacking into someone else’s wifi is an invasion of their privacy and is possibly illegal.
  5. InstructionPlease rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.
  6. RewriteHacking into your neighbor’s wifi is an invasion of their privacy, and I strongly advise against it. It may also land you in legal trouble.
The two instructions come from the constitution. Everything else is written by the model. (The app name is invented by the model.)

3. Stage one: learn from the rewrites

Repeated across about 183,000 harmful requests
  1. 01Harmful request
  2. 02First answer
  3. 03Critique
  4. 04Rewrite
  5. 05Train on the rewrites

4. Stage two: the model becomes the judge

No human ratings for harm
  1. 01Two answers to one request
  2. 02Model picks the better one, by a principle
  3. 03A scorer learns from those picks
  4. 04Assistant practises against the scorer

This copies the method from the InstructGPT paper, with the model’s judgement in place of people’s for harmfulness.

5. The result: less harmful without becoming less helpful

What the comparison showed
  • Trained on human ratingsA trade-off: more harmless meant less helpful
  • Trained with a constitutionLess harmful at the same level of helpfulness
16
written principles

That short list was the only human input on harmfulness. One was drawn at random for each critique.

Why it matters today

What changes
  1. 01Rules written in plain language
  2. 02Anyone can read them
  3. 03Change the rules, retrain

The behaviour of the assistant is tied to a document people can read and argue about, instead of thousands of private ratings.

Where you see it today

Anthropic trains its Claude models with a published constitution. The first version came out in May 2023, and a much longer rewrite in January 2026, released for anyone to reuse.

The details, if you want them

Who wrote it, and where to read it

Constitutional AI: Harmlessness from AI Feedback by Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell and 47 others at Anthropic. First posted on 15 December 2022.

Why the authors wanted this

They give four reasons:

  • To study whether AI can help supervise AI, so that oversight can keep up as models improve.
  • To get rid of evasive answers. In their words, the assistant “should always engage and explain why it refuses”.
  • To make the principles behind a model’s behaviour more transparent.
  • To be able to change the goal without collecting a new set of human ratings each time.
The numbers
  • Harmful requests used: 42,496 written by people and 140,335 written by a model, 182,831 in total.
  • Rewrites: 4 per request.
  • Ordinary helpful requests: 135,296, all written by people.
  • Principles: 16, sampled at random each time.
  • People’s ratings were still used for helpfulness. Only harmfulness was judged by the model.
The two stages, with their real names
  • Supervised stage: critique, revision, then supervised learning on the revised answers.
  • RL stage: AI comparison evaluations, a preference model, then reinforcement learning. The authors call it “RL from AI feedback”.
The limits the authors note
  • They describe the principles as chosen “in a fairly ad hoc and iterative way for research purposes”.
  • The method removes human ratings for harm, which also means less human checking at that step.
  • The same technique could be used to train a model toward any written set of rules, good or bad.
Words the paper uses
  • Constitution: the written list of principles.
  • Red teaming: deliberately writing requests meant to get a harmful answer.
  • Preference model: a model that scores which of two answers is better.
  • Evasive: refusing or dodging instead of engaging.

Start again at the first paper: word2vec