← Papers
5 min read

BERT, in five pictures

The 2018 paper that taught a model language by hiding words and making it guess them, shown as five short animations.

PapersTransformer
Watch it in about a minute

In 2018, four researchers at Google trained a Transformer with a simple game: hide some words, and guess them using everything on both sides of the gap. They called it BERT.

1. Hide a word, guess it from both sides

Tap a word to hide it
Theafterit
The model's top guesses
  • app61%
  • server17%
  • site9%
The model's top guesses
  • crashed48%
  • froze21%
  • broke12%
The model's top guesses
  • updated52%
  • restarted18%
  • launched11%
The model's top guesses
  • first6%
  • company5%
  • man4%
The model's top guesses
  • is22%
  • was15%
  • will9%
The model's top guesses
  • updated52%
  • restarted18%
  • launched11%
Hide "app", then switch to "Left to right only". With just the word "The" to go on, the guesses are poor. Seeing both sides is the whole trick. Made-up numbers to show the idea.

2. The masking recipe

15% of the words are chosen. What happens to each chosen word
  • Replaced with a blank 80%
  • Swapped for a random word 10%
  • Left as it is 10%
The mix stops the model from relying on the blank, which never appears in real use.

3. A second game: does this sentence follow that one?

Half the pairs are real, half are random
A real pair
  1. Sentence Athe man went to [MASK] store
  2. Sentence Bhe bought a gallon [MASK] milk
  3. LabelB really follows A
A random pair
  1. Sentence Athe man [MASK] to the store
  2. Sentence Bpenguin [MASK] are flightless birds
  3. LabelB does not follow A
This teaches the model how sentences relate, which matters for answering questions. The example pairs are the paper's own.

4. Train once, reuse for everything

One model, many jobs
  1. 01Read 3.3 billion words
  2. 02One general model
  3. 03A little extra training
  4. 0411 different tasks

The extra training for each task took at most an hour on one cloud machine.

5. The result: top scores on 11 tasks

Average score on a set of language tests (higher is better)
  • Best before GPT 74
  • GPT (the first one) 75.1
  • BERT, base size 79.6
  • BERT, large size 82.1
The GLUE benchmark, from the paper's comparison table.
Answering questions about a passage (higher is better)
  • People 91.2
  • BERT, one model 91.8
  • BERT, several voting together 93.2
F1 score on the SQuAD v1.1 test.

Why it matters today

What came next
  1. 01BERT, 2018
  2. 02Models that read and understand
  3. 03Search, sorting and embeddings

BERT reads but does not write. Models built on it are still widely used to search, to sort text into categories, and to make embeddings. The search step in the RAG paper uses BERT.

Where you see it today

In October 2019 Google said it had started using BERT in Search, for about one in ten English searches in the US. Its example was “2019 brazil traveler to usa need a visa”: BERT helped Search see that the word “to” matters, so the traveller is going to the US, not leaving it.

The details, if you want them

Who wrote it, and where to read it

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google AI Language. First posted on 11 October 2018.

BERT stands for Bidirectional Encoder Representations from Transformers.

What was wrong with the older models
  • The first GPT read left to right only. In the paper’s words, “every token can only attend to previous tokens”.
  • Another model, ELMo, read in both directions but separately, then glued the two results together. The paper calls this “a shallow concatenation”.
  • BERT’s point is that a truly two-sided model is not possible with the usual “predict the next word” training, because each word could see itself. Hiding words gets around that.
The size of the model
  • Base: 12 layers, 768 numbers per word, 12 attention heads, 110 million parameters.
  • Large: 24 layers, 1,024 numbers per word, 16 heads, 340 million parameters.
  • Words are split into pieces from a list of 30,000.
  • Every input starts with a special [CLS] marker, and [SEP] separates the two sentences.
How it was trained
  • Text: 800 million words from books and 2,500 million words from English Wikipedia.
  • Two games at once: fill in the blanks, and decide whether sentence B follows sentence A. The final model got the second game right 97% to 98% of the time.
  • Then, per task: add one small output layer and train everything a little more.
All the results
  • GLUE leaderboard: 80.5, which was 7.7 points above the previous best.
  • MultiNLI (does one sentence follow from another): 86.7%, up 4.6 points.
  • SQuAD v1.1 (question answering): 93.2 F1, up 1.5 points.
  • SQuAD v2.0 (questions that may have no answer): 83.1 F1, up 5.1 points. People score 89.5 on this one, so BERT was still behind.
  • Base and large BERT beat the earlier best by 4.5 and 7.0 points on average.
Words the paper uses
  • Pre-training: the first, general round of training on plain text.
  • Fine-tuning: the short second round on one specific task.
  • Masked language model: the fill-in-the-blank game. Also called a Cloze task.
  • Next sentence prediction: the does-B-follow-A game.
  • Bidirectional: using the words on both sides.

Next paper: GPT-3