Attention Is All You Need, in eight pictures
The 2017 paper behind today's AI chat models, shown as eight short animations you can watch and poke.
In 2017, eight researchers published a new design for AI models called the Transformer. Almost every chat model today is built on it. You can follow the whole idea from the pictures.
1. Before: one word at a time
5 steps, one after another
2. The idea: look at everything at once
1 step, all at once
3. Attention picks what matters
"it" looks mostly at "app" (64%).
4. How a word decides where to look
1 How well does each word match "it"?2 Scores become shares of 100%
5. Eight heads, eight questions
"it" looks mostly at "app" (64%).
6. Word order comes from a pattern
Word 4 starts with +0.14, -0.99, +0.25, -0.97 ...
7. The whole machine
- Input words + position
- Each piece of a word becomes a list of 512 numbers (an embedding). A position signal is added so the model knows the word order.
- Self-attention
- Every word looks at every other word in the input, in one step, through 8 heads at once. This is the idea the paper is named after.
- Feed-forward
- A small two-layer network applied to each word on its own. It widens the 512 numbers to 2048, keeps the positive part (ReLU), and narrows back to 512.
- Output so far + position
- The words already written, shifted one place to the right, with the same position signal added.
- Masked self-attention
- The output words look at each other, but never at words that come later. So the guess for word 5 can only use words 1 to 4.
- Attention over the encoder
- Each output word looks at every word of the input sentence. The question comes from the decoder; the answers come from the encoder.
- Feed-forward
- The same kind of small network as in the encoder, with its own weights in every layer.
- Next-word probabilities
- A final linear layer and a softmax turn the numbers into a probability for every word in the vocabulary. This layer shares its weights with the two embedding layers.
- Around every box: add and normalise
- Each box adds its input back to its output (a residual connection) and then normalises the result. The paper writes this as LayerNorm(x + Sublayer(x)).
8. The result: better and cheaper
Why it matters today
- 01Transformer, 2017
- 02BERT and GPT
- 03AI chat models
The T in GPT stands for Transformer. Because nothing waits in line, training can be spread over many graphics cards, which is a big part of why models could grow so large.
The details, if you want them
Who wrote it, and where to read it
Attention Is All You Need was first posted on 12 June 2017 and presented at the NIPS 2017 conference.
The authors are Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, mostly at Google Brain and Google Research. The paper says all eight contributed equally and the order of names is random. Their code is in the tensor2tensor library.
What was wrong with the older models
- Recurrent networks read one word at a time, so the work cannot be done in parallel. The paper calls this the “fundamental constraint of sequential computation”.
- Convolutional models (the paper names the Extended Neural GPU, ByteNet and ConvS2S) run in parallel, but two distant words still need several layers before they meet.
- The cost of self-attention: comparing every word with every other word means the work grows with the square of the sentence length. That is still less work than a recurrent layer when the sentence is shorter than 512 words, which is the usual case in translation.
The attention formula
Attention(Q, K, V) = softmax( Q × Kᵀ / √d_k ) × V- Q, K and V are the query (what a word is looking for), the key (what each word offers) and the value (each word’s content).
- Q × Kᵀ gives the match scores from picture 4.
- softmax turns scores into shares of 100%.
- √d_k shrinks the scores first. Without it, large scores push almost all the attention onto one word and the model learns slowly.
d_kis 64 here. - Hiding later words: the half of the model that writes is not allowed to look at words it has not written yet.
The size of the model
- 6 layers in the reading half (the encoder) and 6 in the writing half (the decoder).
- Every word is carried as a list of 512 numbers.
- 8 heads, each working with 64 numbers.
- The small network after each attention step widens to 2048 numbers and narrows back.
- Every part adds its input back to its output and then normalises the result.
- About 65 million parameters in the base model, and about 213 million in the big one.
- The position pattern is made of sine and cosine waves. A learned pattern gave nearly the same score (25.7 against 25.8).
How it was trained
- Data: about 4.5 million English to German sentence pairs, and 36 million English to French sentences, split into tokens.
- Hardware: one machine with 8 NVIDIA P100 graphics cards.
- Time: 12 hours for the base model (100,000 steps) and 3.5 days for the big model (300,000 steps).
- Optimiser: Adam, with the learning rate raised for the first 4,000 steps and then slowly lowered.
- Dropout of 0.1 and label smoothing of 0.1, two standard tricks that stop the model from becoming overconfident.
All the results
Translations are scored with BLEU, which measures overlap with human translations. Higher is better.
- English to German: the big Transformer scored 28.4 and the base model 27.3. The best earlier result was 26.36, from several ConvS2S models voting together.
- English to French: the big Transformer scored 41.8 against 41.29 for the best earlier result. The base model scored 38.1, which did not beat the older models. The paper’s text gives 41.0 in one place and 41.8 in the abstract and table.
- Number of heads: 1 head scored 24.9, 8 heads 25.8, and 32 heads 25.4. Too few and too many both hurt.
- Depth: 2 layers scored 23.7 and 6 layers 25.8.
- No dropout: the score fell from 25.8 to 24.6.
- A different task: on working out the grammar tree of English sentences, a 4-layer Transformer scored 91.3, and 92.7 with extra training data, with very little tuning.
Words the paper uses
- Self-attention: each word looking at the other words in the same sentence.
- Sequence transduction: turning one sequence into another, such as English into German.
- Recurrent network (RNN): a model that reads one word at a time and carries a memory forward.
- Encoder and decoder: the half that reads, and the half that writes.
- Multi-head attention: running several attention steps side by side.
- Positional encoding: the pattern added to each word so the model knows the order.
- d_model: how many numbers represent each word. 512 in the base model.
Next paper: BERT
Related: LLM · Neural network · Token · Embeddings · Context window