← Papers
5 min read

GPT-3, in five pictures

The 2020 paper that showed a big enough model can learn a task from a few examples in the prompt, shown as five short animations.

PapersPrompting
Watch it in about a minute

In 2020, 31 researchers at OpenAI built a model so large that you could teach it a new task just by showing it examples in the prompt. No retraining needed.

1. Before: every task needed its own training

The old way, repeated for every single task
  1. 01Collect thousands of examples
  2. 02Retrain the model
  3. 03It can do one task

2. The idea: put the examples in the prompt

Same model, nothing retrained
No examples
  1. PromptTranslate English to French: cheese =>
  2. Modelfromage
One example
  1. PromptTranslate English to French:
  2. Examplesea otter => loutre de mer
  3. Promptcheese =>
  4. Modelfromage
A few examples
  1. PromptTranslate English to French:
  2. Examplesea otter => loutre de mer
  3. Examplepeppermint => menthe poivrée
  4. Exampleplush giraffe => girafe peluche
  5. Promptcheese =>
  6. Modelfromage
The paper calls these zero-shot, one-shot and few-shot. The model's settings never change. It picks up the task from the pattern in front of it. Made-up example to show the idea.

3. More examples, better answers

Trivia questions answered correctly
  • No examples 64.3%
  • One example 68.0%
  • A few examples 71.2%
With a few examples, it matched the best specially trained models working without a search tool.

4. What made it possible: size

Parameters (the model's adjustable settings)
  • GPT-2, the year before 1.5 billion
  • GPT-3 175 billion
Trained on about 300 billion tokens of text.

5. Where it shone, and where it broke

Adding numbers, from a few examples
  • 2-digit numbers 100%
  • 4-digit numbers 25.5%
  • 5-digit numbers 9.3%
It was never taught arithmetic. It also was not reliable at it.

People asked to spot which news articles GPT-3 had written were right about 52% of the time. Guessing gets 50%.

Why it matters today

What came next
  1. 01Examples in the prompt
  2. 02Prompting as a skill
  3. 03One model for many tasks

This is why writing a good prompt became a skill of its own, and why the context window matters: it is the space your examples have to fit in.

Where you see it today

In June 2020 OpenAI opened GPT-3 to companies through an API it described as “text in, text out”. Early users included Algolia, Quizlet and Reddit. Putting a few examples in the prompt, as this paper did, is still how many teams get a model to follow their format.

The details, if you want them

Who wrote it, and where to read it

Language Models are Few-Shot Learners by Tom B. Brown and 30 others at OpenAI. First posted on 28 May 2020.

The exact definitions
  • Fine-tuning: changing the model’s settings by training on examples for one task. GPT-3 was not fine-tuned in this paper.
  • Few-shot: a few worked examples are placed in the prompt, typically 10 to 100, as many as fit.
  • One-shot: one example, plus a description of the task.
  • Zero-shot: only a description of the task.
  • In all three, the paper stresses that the model’s settings are not updated.
  • The prompt and answer together had to fit in 2,048 tokens.
The size of the model
  • Eight sizes were trained, from 125 million to 175 billion parameters.
  • The largest has 96 layers, 12,288 numbers per token and 96 attention heads.
  • The paper describes it as 10 times larger than any previous model of its kind.
What it was trained on
  • A filtered copy of a large crawl of the web (about 410 billion tokens), which made up 60% of the training mix.
  • A second web collection, two sets of books, and English Wikipedia made up the rest.
  • The text was 93% English.
More results
  • Learning a made-up word: given the sentence ‘A “whatpu” is a small, furry animal native to Tanzania’, the model wrote: “We were traveling in Africa and we saw these very cute whatpus.”
  • Conversation questions (CoQA): 81.5 with no examples and 85.0 with a few. Specially trained models scored 90.7.
  • Translation into English from French and German beat the best earlier results. Translation out of English fell well short.
  • Where it stayed weak: comparing two sentences, and some reading tests. On one word-meaning test it scored 49.4%, the same as guessing.
The limits the authors list
  • Long passages start to repeat themselves and lose the thread.
  • It struggles with everyday physics questions.
  • It sees far more text in training than a person reads in a lifetime.
  • It is expensive to run.
  • It keeps the biases of its training text. In one test, 83% of 388 job titles were more likely to be followed by a male word.
  • A bug meant some test material may have appeared in the training text, and retraining was too costly to fix it.
Words the paper uses
  • In-context learning: picking up a task from the examples in the prompt.
  • Shot: one worked example.
  • Parameters: the adjustable settings inside a model. See parameters.
  • Token: a piece of a word. See token.

Next paper: RAG