GPT-3, in five pictures
The 2020 paper that showed a big enough model can learn a task from a few examples in the prompt, shown as five short animations.
In 2020, 31 researchers at OpenAI built a model so large that you could teach it a new task just by showing it examples in the prompt. No retraining needed.
1. Before: every task needed its own training
- 01Collect thousands of examples
- 02Retrain the model
- 03It can do one task
2. The idea: put the examples in the prompt
- PromptTranslate English to French: cheese =>
- Modelfromage
- PromptTranslate English to French:
- Examplesea otter => loutre de mer
- Promptcheese =>
- Modelfromage
- PromptTranslate English to French:
- Examplesea otter => loutre de mer
- Examplepeppermint => menthe poivrée
- Exampleplush giraffe => girafe peluche
- Promptcheese =>
- Modelfromage
3. More examples, better answers
4. What made it possible: size
5. Where it shone, and where it broke
People asked to spot which news articles GPT-3 had written were right about 52% of the time. Guessing gets 50%.
Why it matters today
- 01Examples in the prompt
- 02Prompting as a skill
- 03One model for many tasks
This is why writing a good prompt became a skill of its own, and why the context window matters: it is the space your examples have to fit in.
In June 2020 OpenAI opened GPT-3 to companies through an API it described as “text in, text out”. Early users included Algolia, Quizlet and Reddit. Putting a few examples in the prompt, as this paper did, is still how many teams get a model to follow their format.
The details, if you want them
Who wrote it, and where to read it
Language Models are Few-Shot Learners by Tom B. Brown and 30 others at OpenAI. First posted on 28 May 2020.
The exact definitions
- Fine-tuning: changing the model’s settings by training on examples for one task. GPT-3 was not fine-tuned in this paper.
- Few-shot: a few worked examples are placed in the prompt, typically 10 to 100, as many as fit.
- One-shot: one example, plus a description of the task.
- Zero-shot: only a description of the task.
- In all three, the paper stresses that the model’s settings are not updated.
- The prompt and answer together had to fit in 2,048 tokens.
The size of the model
- Eight sizes were trained, from 125 million to 175 billion parameters.
- The largest has 96 layers, 12,288 numbers per token and 96 attention heads.
- The paper describes it as 10 times larger than any previous model of its kind.
What it was trained on
- A filtered copy of a large crawl of the web (about 410 billion tokens), which made up 60% of the training mix.
- A second web collection, two sets of books, and English Wikipedia made up the rest.
- The text was 93% English.
More results
- Learning a made-up word: given the sentence ‘A “whatpu” is a small, furry animal native to Tanzania’, the model wrote: “We were traveling in Africa and we saw these very cute whatpus.”
- Conversation questions (CoQA): 81.5 with no examples and 85.0 with a few. Specially trained models scored 90.7.
- Translation into English from French and German beat the best earlier results. Translation out of English fell well short.
- Where it stayed weak: comparing two sentences, and some reading tests. On one word-meaning test it scored 49.4%, the same as guessing.
The limits the authors list
- Long passages start to repeat themselves and lose the thread.
- It struggles with everyday physics questions.
- It sees far more text in training than a person reads in a lifetime.
- It is expensive to run.
- It keeps the biases of its training text. In one test, 83% of 388 job titles were more likely to be followed by a male word.
- A bug meant some test material may have appeared in the training text, and retraining was too costly to fix it.
Words the paper uses
- In-context learning: picking up a task from the examples in the prompt.
- Shot: one worked example.
- Parameters: the adjustable settings inside a model. See parameters.
- Token: a piece of a word. See token.
Next paper: RAG