InstructGPT, in six pictures
The 2022 paper that used human feedback to turn a text predictor into an assistant that follows instructions, in six short animations.
In 2022, twenty researchers at OpenAI took GPT-3, which was built to continue text, and used feedback from people to teach it to do what it is asked. The result was called InstructGPT.
1. The problem: predicting text is not the same as helping
- YouExplain the moon landing to a 6 year old in a few sentences.
- ModelExplain the theory of gravity to a 6 year old. Explain the theory of relativity to a 6 year old in a few sentences.
- YouExplain the moon landing to a 6 year old in a few sentences.
- ModelPeople went to the moon, took pictures of what they saw, and sent them back to Earth so we could all see them.
2. The method: three steps
- 1People write good answers — about 13,000 prompts, and the model learns to imitate them
- 2People rank the model’s answers — about 33,000 prompts, used to train a second model that scores answers
- 3The model practises against that scorer — about 31,000 more prompts, with no people needed
3. What ranking looks like
- PromptWrite a one-line thank-you note to a customer.
- 1stThank you for your order. We are glad you chose us.
- 2ndThanks.
- 3rdWrite a one-line apology note to a customer.
4. The result: people preferred it, even at 1% of the size
5. It cost almost nothing next to the original training
6. What it did not fix
- Goes along with a false premise"Why is it important to eat socks after meditating?"
- Hedges too muchEven when there is a clear answer
- Still follows harmful instructionsIn most cases
- BiasNo real improvement over GPT-3
Why it matters today
- 01Text predictor
- 02Human feedback
- 03Chat assistants
This method is usually called RLHF, reinforcement learning from human feedback. It is how language models became assistants you can simply ask. The Constitutional AI paper later tried replacing much of the human feedback.
When OpenAI launched ChatGPT in November 2022, it described it as a sibling model to InstructGPT, trained with the same human-feedback method.
The details, if you want them
Who wrote it, and where to read it
Training language models to follow instructions with human feedback by Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin and 14 others at OpenAI. First posted on 4 March 2022.
The problem, in the paper’s words
- A language model is trained to predict “the next token on a webpage from the internet”.
- What people want is for it to “follow the user’s instructions helpfully and safely”.
- Because those two goals differ, the authors call the original training goal “misaligned”.
- They want models that are helpful, honest and harmless.
The three steps, with their real names
- Supervised fine-tuning: the model learns from answers written by people.
- Reward model: a second model learns to predict which answer people would rank higher.
- PPO: a standard practice-and-improve method, used to push the first model toward answers the reward model scores highly.
- Three sizes were trained: 1.3 billion, 6 billion and 175 billion parameters.
More results
- Against GPT-3 given a few examples in the prompt, InstructGPT was still preferred 71% of the time.
- People who had not taken part in the training preferred InstructGPT at about the same rate, so it had not just learned the tastes of its own labelers.
- When asked to be respectful, it produced about 25% fewer toxic outputs than GPT-3. Without that request the gap disappeared, and when asked to be toxic it was worse than GPT-3.
- It followed instructions in other languages and about code, although almost none of its feedback covered those.
- Tuning for helpfulness made some standard test scores drop. The authors call this an “alignment tax” and reduced it by mixing in some of the original training.
Whose preferences?
The authors are direct about this. The model is aligned to the preferences of about 40 contractors, the researchers who wrote their instructions, and the customers whose prompts were used. Most of the contractors were English-speaking, and most comparisons were judged by a single person. In the paper’s words, the model is “neither fully aligned nor fully safe”.
Words the paper uses
- Alignment: getting a model to do what people actually want.
- RLHF: reinforcement learning from human feedback, the three-step method above.
- Labelers: the people who wrote and ranked answers.
- Hallucination: stating something that is not in the source or not true. See hallucination.
Next paper: ReAct