← Papers
6 min read

Attention Is All You Need, in eight pictures

The 2017 paper behind today's AI chat models, shown as eight short animations you can watch and poke.

PapersTransformer
Watch it in about a minute

In 2017, eight researchers published a new design for AI models called the Transformer. Almost every chat model today is built on it. You can follow the whole idea from the pictures.

1. Before: one word at a time

Word by word
The app crashed after it updated

5 steps, one after another

Older models read a sentence word by word. Each step has to wait for the one before, and by the end the first words have faded.

2. The idea: look at everything at once

All at once
The app crashed after it updated

1 step, all at once

The paper throws out word-by-word reading. Every word connects to every other word in a single step. Nothing waits, and nothing fades.

3. Attention picks what matters

Tap a word

"it" looks mostly at "app" (64%).

Thicker line, more attention. "it" looks hardest at "app", because that is what "it" means here. Tap another word. Made-up numbers to show the idea.

4. How a word decides where to look

From scores to shares
0.03%
3.064%
1.211%
0.03%
1.09%
1.211%
The app crashed after it updated

1 How well does each word match "it"?2 Scores become shares of 100%

Each word gets a score for how well it matches what "it" is looking for. The scores are then turned into shares of 100%, and the best match takes most of it. Made-up numbers to show the idea.

5. Eight heads, eight questions

Tap a head, then a word

"it" looks mostly at "app" (64%).

The model does this 8 times side by side. Each copy, called a head, can look for something different. Three examples are shown. Made-up numbers to show the idea.

6. Word order comes from a pattern

Looking at everything at once loses the order. So every position gets its own pattern.

Word 4 starts with +0.14, -0.99, +0.25, -0.97 ...

Green = positive, grey = negative. Fast waves on the left, slow on the right. 32 of the first 128 numbers shown; the other 384 change even more slowly.

7. The whole machine

One half reads, the other half writes. Tap a part.
Encoder · reads the input sentence
x 6 layers
Decoder · writes the output, one word at a time
x 6 layers
Input words + position
Each piece of a word becomes a list of 512 numbers (an embedding). A position signal is added so the model knows the word order.
Self-attention
Every word looks at every other word in the input, in one step, through 8 heads at once. This is the idea the paper is named after.
Feed-forward
A small two-layer network applied to each word on its own. It widens the 512 numbers to 2048, keeps the positive part (ReLU), and narrows back to 512.
Output so far + position
The words already written, shifted one place to the right, with the same position signal added.
Masked self-attention
The output words look at each other, but never at words that come later. So the guess for word 5 can only use words 1 to 4.
Attention over the encoder
Each output word looks at every word of the input sentence. The question comes from the decoder; the answers come from the encoder.
Feed-forward
The same kind of small network as in the encoder, with its own weights in every layer.
Next-word probabilities
A final linear layer and a softmax turn the numbers into a probability for every word in the vocabulary. This layer shares its weights with the two embedding layers.
Around every box: add and normalise
Each box adds its input back to its output (a residual connection) and then normalises the result. The paper writes this as LayerNorm(x + Sublayer(x)).
Read each column from the bottom up.

8. The result: better and cheaper

English to German translation score (higher is better)
  • Best earlier model 26.36
  • Transformer 28.4
The earlier best was several models voting together. The Transformer is one model.
Training cost (lower is better)
  • Transformer, small version 1×
  • ConvS2S 2.9×
  • MoE 6.1×
  • GNMT + RL 7×
Even the small Transformer beat all three on score. It trained in 12 hours on 8 graphics cards.

Why it matters today

What came next
  1. 01Transformer, 2017
  2. 02BERT and GPT
  3. 03AI chat models

The T in GPT stands for Transformer. Because nothing waits in line, training can be spread over many graphics cards, which is a big part of why models could grow so large.

The details, if you want them

Who wrote it, and where to read it

Attention Is All You Need was first posted on 12 June 2017 and presented at the NIPS 2017 conference.

The authors are Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin, mostly at Google Brain and Google Research. The paper says all eight contributed equally and the order of names is random. Their code is in the tensor2tensor library.

What was wrong with the older models
  • Recurrent networks read one word at a time, so the work cannot be done in parallel. The paper calls this the “fundamental constraint of sequential computation”.
  • Convolutional models (the paper names the Extended Neural GPU, ByteNet and ConvS2S) run in parallel, but two distant words still need several layers before they meet.
  • The cost of self-attention: comparing every word with every other word means the work grows with the square of the sentence length. That is still less work than a recurrent layer when the sentence is shorter than 512 words, which is the usual case in translation.
The attention formula
Attention(Q, K, V) = softmax( Q × Kᵀ / √d_k ) × V
  • Q, K and V are the query (what a word is looking for), the key (what each word offers) and the value (each word’s content).
  • Q × Kᵀ gives the match scores from picture 4.
  • softmax turns scores into shares of 100%.
  • √d_k shrinks the scores first. Without it, large scores push almost all the attention onto one word and the model learns slowly. d_k is 64 here.
  • Hiding later words: the half of the model that writes is not allowed to look at words it has not written yet.
The size of the model
  • 6 layers in the reading half (the encoder) and 6 in the writing half (the decoder).
  • Every word is carried as a list of 512 numbers.
  • 8 heads, each working with 64 numbers.
  • The small network after each attention step widens to 2048 numbers and narrows back.
  • Every part adds its input back to its output and then normalises the result.
  • About 65 million parameters in the base model, and about 213 million in the big one.
  • The position pattern is made of sine and cosine waves. A learned pattern gave nearly the same score (25.7 against 25.8).
How it was trained
  • Data: about 4.5 million English to German sentence pairs, and 36 million English to French sentences, split into tokens.
  • Hardware: one machine with 8 NVIDIA P100 graphics cards.
  • Time: 12 hours for the base model (100,000 steps) and 3.5 days for the big model (300,000 steps).
  • Optimiser: Adam, with the learning rate raised for the first 4,000 steps and then slowly lowered.
  • Dropout of 0.1 and label smoothing of 0.1, two standard tricks that stop the model from becoming overconfident.
All the results

Translations are scored with BLEU, which measures overlap with human translations. Higher is better.

  • English to German: the big Transformer scored 28.4 and the base model 27.3. The best earlier result was 26.36, from several ConvS2S models voting together.
  • English to French: the big Transformer scored 41.8 against 41.29 for the best earlier result. The base model scored 38.1, which did not beat the older models. The paper’s text gives 41.0 in one place and 41.8 in the abstract and table.
  • Number of heads: 1 head scored 24.9, 8 heads 25.8, and 32 heads 25.4. Too few and too many both hurt.
  • Depth: 2 layers scored 23.7 and 6 layers 25.8.
  • No dropout: the score fell from 25.8 to 24.6.
  • A different task: on working out the grammar tree of English sentences, a 4-layer Transformer scored 91.3, and 92.7 with extra training data, with very little tuning.
Words the paper uses
  • Self-attention: each word looking at the other words in the same sentence.
  • Sequence transduction: turning one sequence into another, such as English into German.
  • Recurrent network (RNN): a model that reads one word at a time and carries a memory forward.
  • Encoder and decoder: the half that reads, and the half that writes.
  • Multi-head attention: running several attention steps side by side.
  • Positional encoding: the pattern added to each word so the model knows the order.
  • d_model: how many numbers represent each word. 512 in the base model.

Next paper: BERT

Related: LLM · Neural network · Token · Embeddings · Context window