BERT, in five pictures
The 2018 paper that taught a model language by hiding words and making it guess them, shown as five short animations.
In 2018, four researchers at Google trained a Transformer with a simple game: hide some words, and guess them using everything on both sides of the gap. They called it BERT.
1. Hide a word, guess it from both sides
2. The masking recipe
3. A second game: does this sentence follow that one?
- Sentence Athe man went to [MASK] store
- Sentence Bhe bought a gallon [MASK] milk
- LabelB really follows A
- Sentence Athe man [MASK] to the store
- Sentence Bpenguin [MASK] are flightless birds
- LabelB does not follow A
4. Train once, reuse for everything
- 01Read 3.3 billion words
- 02One general model
- 03A little extra training
- 0411 different tasks
The extra training for each task took at most an hour on one cloud machine.
5. The result: top scores on 11 tasks
Why it matters today
- 01BERT, 2018
- 02Models that read and understand
- 03Search, sorting and embeddings
BERT reads but does not write. Models built on it are still widely used to search, to sort text into categories, and to make embeddings. The search step in the RAG paper uses BERT.
In October 2019 Google said it had started using BERT in Search, for about one in ten English searches in the US. Its example was “2019 brazil traveler to usa need a visa”: BERT helped Search see that the word “to” matters, so the traveller is going to the US, not leaving it.
The details, if you want them
Who wrote it, and where to read it
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova at Google AI Language. First posted on 11 October 2018.
BERT stands for Bidirectional Encoder Representations from Transformers.
What was wrong with the older models
- The first GPT read left to right only. In the paper’s words, “every token can only attend to previous tokens”.
- Another model, ELMo, read in both directions but separately, then glued the two results together. The paper calls this “a shallow concatenation”.
- BERT’s point is that a truly two-sided model is not possible with the usual “predict the next word” training, because each word could see itself. Hiding words gets around that.
The size of the model
- Base: 12 layers, 768 numbers per word, 12 attention heads, 110 million parameters.
- Large: 24 layers, 1,024 numbers per word, 16 heads, 340 million parameters.
- Words are split into pieces from a list of 30,000.
- Every input starts with a special [CLS] marker, and [SEP] separates the two sentences.
How it was trained
- Text: 800 million words from books and 2,500 million words from English Wikipedia.
- Two games at once: fill in the blanks, and decide whether sentence B follows sentence A. The final model got the second game right 97% to 98% of the time.
- Then, per task: add one small output layer and train everything a little more.
All the results
- GLUE leaderboard: 80.5, which was 7.7 points above the previous best.
- MultiNLI (does one sentence follow from another): 86.7%, up 4.6 points.
- SQuAD v1.1 (question answering): 93.2 F1, up 1.5 points.
- SQuAD v2.0 (questions that may have no answer): 83.1 F1, up 5.1 points. People score 89.5 on this one, so BERT was still behind.
- Base and large BERT beat the earlier best by 4.5 and 7.0 points on average.
Words the paper uses
- Pre-training: the first, general round of training on plain text.
- Fine-tuning: the short second round on one specific task.
- Masked language model: the fill-in-the-blank game. Also called a Cloze task.
- Next sentence prediction: the does-B-follow-A game.
- Bidirectional: using the words on both sides.
Next paper: GPT-3