← Papers
5 min read

word2vec, in five pictures

The 2013 paper that turned words into numbers a computer can compare, shown as five short animations.

PapersEmbeddings
Watch it in about a minute

In 2013, four researchers at Google showed a fast way to give every word a list of numbers, so that words with similar meanings get similar numbers. Those lists are what we now call embeddings.

1. Before: a word was just an ID

How older systems saw words (made-up IDs)
  • catword #4,521
  • kittenword #18,007
  • carburettorword #4,522

To the computer, “cat” was no closer to “kitten” than to “carburettor”. The paper’s opening line says systems “treat words as atomic units”.

2. The idea: learn a word from its neighbours

Tap a word to move the window

Use "The", "app", "after", "it" to guess the hidden word.

The model plays a guessing game over billions of words. Words that show up next to the same neighbours end up with similar numbers. The paper tries two versions of the game.

3. What comes out: a map of meaning

Similar words land close together (2 of the hundreds of numbers)
Animals Cities Size cat kitten dog Paris Rome Tokyo big bigger biggest
Made-up positions to show the idea

4. The surprise: you can do arithmetic with meaning

Arithmetic with meaning
manwomankingqueen

king − man + woman ≈ queen

FranceParisItalyRome

Italy − France + Paris ≈ Rome

bigbiggestsmallsmallest

small − big + biggest ≈ smallest

The step from one word to another is a direction on the map. Copy that step onto a different word and you land on the matching answer. All three examples are from the paper.

5. The result: better and far cheaper

Word-relationship test, % correct (higher is better)
  • Older style of model 50.8
  • New: guess the middle word 63.7
  • New: guess the neighbours 65.6
All three trained on the same 6 billion words of news text.
Computer time to train (lower is better)
  • Older style of model 14 days × 180 cores
  • New: guess the middle word 2 days × 140 cores
  • New: guess the neighbours 2.5 days × 125 cores
The paper says the core counts are estimates.

Why it matters today

What came next
  1. 01Words as numbers, 2013
  2. 02Sentences as numbers
  3. 03Search by meaning and RAG

Every time an AI tool searches by meaning instead of exact words, it is using this idea. See embeddings and RAG.

Where you see it today

Airbnb used word2vec’s method on holiday listings instead of words: listings people clicked in the same search session were treated like neighbouring words. The resulting map of similar listings helped rank search results and recommend similar homes (described in a 2018 paper).

The details, if you want them

Who wrote it, and where to read it

Efficient Estimation of Word Representations in Vector Space by Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean, all at Google. First posted on 16 January 2013.

What was wrong with the older models
  • Words were stored as positions in a vocabulary list, with no idea of similarity between them.
  • Earlier neural models did learn word numbers, but they were slow. The paper says none had been trained on more than a few hundred million words, with only 50 to 100 numbers per word.
  • One of the older models in the comparison took about 8 weeks to train on a single processor.
The two guessing games
  • CBOW (continuous bag-of-words): hide the middle word and guess it from the words around it. The paper’s best setup used four words before and four after. Word order inside the window is ignored.
  • Skip-gram: take one word and guess the words around it, up to a set distance. Nearby words are used more often than distant ones.
  • Both are deliberately simple. They remove the expensive middle layer that older models had, which is where the speed comes from.
How the arithmetic is checked
  • To answer “big is to biggest as small is to what?”, the model computes biggest − big + small and looks for the closest word, measured by cosine distance.
  • The answer only counts if the closest word is exactly right. Synonyms count as wrong.
  • The test has 19,544 questions: 8,869 about meaning (capital cities, currencies, man and woman) and 10,675 about grammar (plurals, past tense, opposites).
All the results
  • On 6 billion words with 1,000 numbers per word, guess-the-neighbours scored 65.6% and guess-the-middle-word 63.7%. The older style of model, with 100 numbers per word, scored 50.8%.
  • Guess-the-neighbours was much better on meaning questions (66.1%), and guess-the-middle-word slightly better on grammar questions (68.9%).
  • On one ordinary processor, guess-the-middle-word trained on 1.6 billion words in about 0.6 days. That is the “less than a day” in the paper’s summary. Guess-the-neighbours took about 2 days on the same data.
  • More data and more numbers per word both helped, with shrinking returns.
Where it goes wrong
  • The paper’s own table of examples includes misses: asked for the chemical symbol pattern (copper to Cu), it paired uranium with plutonium. Asked for national foods, it paired France with tapas.
  • The authors estimate only about 60% of that table would pass their strict scoring.
  • The model knows nothing about how words are built (walk, walked, walking), which the authors say limits grammar questions.
Words the paper uses
  • Word vector: the list of numbers that stands for a word. Today usually called an embedding.
  • Vector space: the “map” all those lists live in.
  • CBOW and Skip-gram: the two guessing games.
  • Cosine distance: a way to measure how closely two lists of numbers point in the same direction.

Next paper: Attention Is All You Need