← Papers
5 min read

The RAG paper, in five pictures

The 2020 paper that let a model look things up before answering, shown as five short animations.

PapersRAG
Watch it in about a minute

In 2020, twelve researchers, most of them at Facebook AI Research, gave a language model a library to search. They named the method retrieval-augmented generation, or RAG.

1. Before: everything had to be memorised

A model that only knows what is inside it
  • New factsCannot be added without retraining
  • SourcesCannot show where an answer came from
  • GapsMay fill them with made-up facts

2. The idea: search first, then answer

One question, start to finish
Your questionWhat is the middle ear?
Your documents, split into chunks
  • The middle ear is the part of the ear behind the eardrum.
  • It contains three small bones called the ossicles.
  • The outer ear collects sound waves.
  • Bats find their way using echoes.
  • A piano has 88 keys.
What the model actually receives
  • Answer using only the notes below.
  • The middle ear is the part of the ear behind the eardrum.
  • It contains three small bones called the ossicles.
  • What is the middle ear?
AnswerThe middle ear sits behind the eardrum and holds three small bones, the ossicles.

The paper used all of Wikipedia as the library: 21 million short passages of 100 words each.

3. The same question, with and without the library

Asked to define the middle ear
Model alone
  1. Questiondefine middle ear
  2. AnswerThe middle ear is the part of the ear between the middle ear and the nose.
With the library
  1. Questiondefine middle ear
  2. AnswerThe middle ear includes the tympanic cavity and the three ossicles.
Both answers are real outputs printed in the paper. The model working alone wrote something fluent and wrong.

4. Update the library, update the answers

Questions about world leaders answered correctly
  • 2016 library, 2016 leaders 70%
  • 2018 library, 2018 leaders 68%
  • 2018 library, 2016 leaders 12%
  • 2016 library, 2018 leaders 4%
Swapping the Wikipedia copy changed what the model knew. The model itself was not retrained.

5. The result: more right answers, fewer made-up ones

Open questions answered exactly right (higher is better)
  • Large model, memory only 34.5
  • Earlier search-based systems 41.5
  • RAG 44.5
The Natural Questions test. The memory-only model was about 17 times larger than RAG.
People judging which answer was more factual
  • The model alone was better 7.1%
  • RAG was better 42.7%
For the rest, judges rated both the same or could not agree.

Why it matters today

What came next
  1. 01RAG, 2020
  2. 02Chat over your own documents
  3. 03Answers that show sources

Most “chat with your documents” tools work this way. The everyday version is explained in What is RAG?.

Where you see it today

Intercom’s Fin, a customer service AI agent, first searches a company’s help centre articles, PDFs and past conversations, then writes its reply from what it found. Intercom describes it as a retrieval-augmented generation design.

The details, if you want them

Who wrote it, and where to read it

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel and Douwe Kiela. The authors were at Facebook AI Research, University College London and New York University. First posted on 22 May 2020 and accepted at NeurIPS 2020.

The two kinds of memory
  • Parametric memory: what the model has stored in its own settings. Here that is a text-writing model called BART, with about 400 million parameters.
  • Non-parametric memory: the library. Here that is Wikipedia from December 2018, turned into embeddings so it can be searched by meaning.
  • The retriever: two copies of BERT. One turns the question into numbers, the other has already done the same for every passage. The closest passages win.
The two versions
  • RAG-Sequence picks its passages once and writes the whole answer from them.
  • RAG-Token can lean on a different passage for each word it writes.
  • Both were trained without anyone marking which passages were the right ones. The system worked that out from whether the final answers were correct.
All the results
  • Open questions, exactly right: 44.5 on Natural Questions, 45.5 on WebQuestions and 52.2 on CuratedTrec, the best published at the time on each.
  • Writing quiz clues: judges found RAG more specific than the model alone in 37.4% of cases, against 16.8% the other way.
  • Checking claims as true or false: 72.5% on a three-way test, within about 4 points of systems built specially for it.
  • Answering without the answer in hand: RAG still got 11.8% right when none of the passages it found contained the answer.
  • Search matters: a learned search beat plain keyword search on every task except fact-checking.
The limits the authors list
  • The library is not perfect. In their words, Wikipedia “will probably never be entirely factual and completely devoid of bias”.
  • The searchable index needed about 100 GB of memory, or 36 GB when compressed.
  • The same technology could be used to produce misleading content.
Words the paper uses
  • Retrieval: searching the library.
  • Generation: writing the answer.
  • Hallucination: a fluent answer that is not true. See hallucination.
  • Provenance: being able to say where a fact came from.

Next paper: LoRA