word2vec, in five pictures
The 2013 paper that turned words into numbers a computer can compare, shown as five short animations.
In 2013, four researchers at Google showed a fast way to give every word a list of numbers, so that words with similar meanings get similar numbers. Those lists are what we now call embeddings.
1. Before: a word was just an ID
- catword #4,521
- kittenword #18,007
- carburettorword #4,522
To the computer, “cat” was no closer to “kitten” than to “carburettor”. The paper’s opening line says systems “treat words as atomic units”.
2. The idea: learn a word from its neighbours
Use "The", "app", "after", "it" to guess the hidden word.
3. What comes out: a map of meaning
4. The surprise: you can do arithmetic with meaning
king − man + woman ≈ queen
Italy − France + Paris ≈ Rome
small − big + biggest ≈ smallest
5. The result: better and far cheaper
Why it matters today
- 01Words as numbers, 2013
- 02Sentences as numbers
- 03Search by meaning and RAG
Every time an AI tool searches by meaning instead of exact words, it is using this idea. See embeddings and RAG.
Airbnb used word2vec’s method on holiday listings instead of words: listings people clicked in the same search session were treated like neighbouring words. The resulting map of similar listings helped rank search results and recommend similar homes (described in a 2018 paper).
The details, if you want them
Who wrote it, and where to read it
Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, all at Google. First posted on 16 January 2013.
What was wrong with the older models
- Words were stored as positions in a vocabulary list, with no idea of similarity between them.
- Earlier neural models did learn word numbers, but they were slow. The paper says none had been trained on more than a few hundred million words, with only 50 to 100 numbers per word.
- One of the older models in the comparison took about 8 weeks to train on a single processor.
The two guessing games
- CBOW (continuous bag-of-words): hide the middle word and guess it from the words around it. The paper’s best setup used four words before and four after. Word order inside the window is ignored.
- Skip-gram: take one word and guess the words around it, up to a set distance. Nearby words are used more often than distant ones.
- Both are deliberately simple. They remove the expensive middle layer that older models had, which is where the speed comes from.
How the arithmetic is checked
- To answer “big is to biggest as small is to what?”, the model computes biggest − big + small and looks for the closest word, measured by cosine distance.
- The answer only counts if the closest word is exactly right. Synonyms count as wrong.
- The test has 19,544 questions: 8,869 about meaning (capital cities, currencies, man and woman) and 10,675 about grammar (plurals, past tense, opposites).
All the results
- On 6 billion words with 1,000 numbers per word, guess-the-neighbours scored 65.6% and guess-the-middle-word 63.7%. The older style of model, with 100 numbers per word, scored 50.8%.
- Guess-the-neighbours was much better on meaning questions (66.1%), and guess-the-middle-word slightly better on grammar questions (68.9%).
- On one ordinary processor, guess-the-middle-word trained on 1.6 billion words in about 0.6 days. That is the “less than a day” in the paper’s summary. Guess-the-neighbours took about 2 days on the same data.
- More data and more numbers per word both helped, with shrinking returns.
Where it goes wrong
- The paper’s own table of examples includes misses: asked for the chemical symbol pattern (copper to Cu), it paired uranium with plutonium. Asked for national foods, it paired France with tapas.
- The authors estimate only about 60% of that table would pass their strict scoring.
- The model knows nothing about how words are built (walk, walked, walking), which the authors say limits grammar questions.
Words the paper uses
- Word vector: the list of numbers that stands for a word. Today usually called an embedding.
- Vector space: the “map” all those lists live in.
- CBOW and Skip-gram: the two guessing games.
- Cosine distance: a way to measure how closely two lists of numbers point in the same direction.
Next paper: Attention Is All You Need