← Papers
6 min read

InstructGPT, in six pictures

The 2022 paper that used human feedback to turn a text predictor into an assistant that follows instructions, in six short animations.

PapersTraining
Watch it in about a minute

In 2022, twenty researchers at OpenAI took GPT-3, which was built to continue text, and used feedback from people to teach it to do what it is asked. The result was called InstructGPT.

1. The problem: predicting text is not the same as helping

The same request, before and after
Before
  1. YouExplain the moon landing to a 6 year old in a few sentences.
  2. ModelExplain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences.
After
  1. YouExplain the moon landing to a 6 year old in a few sentences.
  2. ModelPeople went to the moon, took pictures of what they saw, and sent them back to Earth so we could all see them.
The first model treats your request as a piece of text to continue, so it writes more requests. The second one answers. Made-up example to show the idea.

2. The method: three steps

Training with human feedback
  1. 1People write good answers — about 13,000 prompts, and the model learns to imitate them
  2. 2People rank the model’s answers — about 33,000 prompts, used to train a second model that scores answers
  3. 3The model practises against that scorer — about 31,000 more prompts, with no people needed

3. What ranking looks like

One prompt, several answers, ranked by a person
  1. PromptWrite a one-line thank-you note to a customer.
  2. 1stThank you for your order. We are glad you chose us.
  3. 2ndThanks.
  4. 3rdWrite a one-line apology note to a customer.
Each person ranked between 4 and 9 answers per prompt. About 40 people did all the writing and ranking. Made-up example to show the idea.

4. The result: people preferred it, even at 1% of the size

Which answer people preferred, same-size models
  • GPT-3 15%
  • InstructGPT 85%
A version of InstructGPT 100 times smaller than GPT-3 was also preferred over it.
Made-up facts when summarising or answering from a given text (lower is better)
  • GPT-3 41%
  • InstructGPT 21%

5. It cost almost nothing next to the original training

Computing used
  • Training GPT-3 in the first place 3,640
  • Step 3, practising against the scorer 60
  • Step 1, imitating people 4.9
In petaflop/s-days, a measure of computing work. The paper's point: feedback helped more than making the model 100 times bigger.

6. What it did not fix

Limits the authors found
  • Goes along with a false premise"Why is it important to eat socks after meditating?"
  • Hedges too muchEven when there is a clear answer
  • Still follows harmful instructionsIn most cases
  • BiasNo real improvement over GPT-3

Why it matters today

What came next
  1. 01Text predictor
  2. 02Human feedback
  3. 03Chat assistants

This method is usually called RLHF, reinforcement learning from human feedback. It is how language models became assistants you can simply ask. The Constitutional AI paper later tried replacing much of the human feedback.

Where you see it today

When OpenAI launched ChatGPT in November 2022, it described it as a sibling model to InstructGPT, trained with the same human-feedback method.

The details, if you want them

Who wrote it, and where to read it

Training language models to follow instructions with human feedback by Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin and 14 others at OpenAI. First posted on 4 March 2022.

The problem, in the paper’s words
  • A language model is trained to predict “the next token on a webpage from the internet”.
  • What people want is for it to “follow the user’s instructions helpfully and safely”.
  • Because those two goals differ, the authors call the original training goal “misaligned”.
  • They want models that are helpful, honest and harmless.
The three steps, with their real names
  • Supervised fine-tuning: the model learns from answers written by people.
  • Reward model: a second model learns to predict which answer people would rank higher.
  • PPO: a standard practice-and-improve method, used to push the first model toward answers the reward model scores highly.
  • Three sizes were trained: 1.3 billion, 6 billion and 175 billion parameters.
More results
  • Against GPT-3 given a few examples in the prompt, InstructGPT was still preferred 71% of the time.
  • People who had not taken part in the training preferred InstructGPT at about the same rate, so it had not just learned the tastes of its own labelers.
  • When asked to be respectful, it produced about 25% fewer toxic outputs than GPT-3. Without that request the gap disappeared, and when asked to be toxic it was worse than GPT-3.
  • It followed instructions in other languages and about code, although almost none of its feedback covered those.
  • Tuning for helpfulness made some standard test scores drop. The authors call this an “alignment tax” and reduced it by mixing in some of the original training.
Whose preferences?

The authors are direct about this. The model is aligned to the preferences of about 40 contractors, the researchers who wrote their instructions, and the customers whose prompts were used. Most of the contractors were English-speaking, and most comparisons were judged by a single person. In the paper’s words, the model is “neither fully aligned nor fully safe”.

Words the paper uses
  • Alignment: getting a model to do what people actually want.
  • RLHF: reinforcement learning from human feedback, the three-step method above.
  • Labelers: the people who wrote and ranked answers.
  • Hallucination: stating something that is not in the source or not true. See hallucination.

Next paper: ReAct