The RAG paper, in five pictures
The 2020 paper that let a model look things up before answering, shown as five short animations.
In 2020, twelve researchers, most of them at Facebook AI Research, gave a language model a library to search. They named the method retrieval-augmented generation, or RAG.
1. Before: everything had to be memorised
- New factsCannot be added without retraining
- SourcesCannot show where an answer came from
- GapsMay fill them with made-up facts
2. The idea: search first, then answer
- The middle ear is the part of the ear behind the eardrum.
- It contains three small bones called the ossicles.
- The outer ear collects sound waves.
- Bats find their way using echoes.
- A piano has 88 keys.
- Answer using only the notes below.
- The middle ear is the part of the ear behind the eardrum.
- It contains three small bones called the ossicles.
- What is the middle ear?
The paper used all of Wikipedia as the library: 21 million short passages of 100 words each.
3. The same question, with and without the library
- Questiondefine middle ear
- AnswerThe middle ear is the part of the ear between the middle ear and the nose.
- Questiondefine middle ear
- AnswerThe middle ear includes the tympanic cavity and the three ossicles.
4. Update the library, update the answers
5. The result: more right answers, fewer made-up ones
Why it matters today
- 01RAG, 2020
- 02Chat over your own documents
- 03Answers that show sources
Most “chat with your documents” tools work this way. The everyday version is explained in What is RAG?.
Intercom’s Fin, a customer service AI agent, first searches a company’s help centre articles, PDFs and past conversations, then writes its reply from what it found. Intercom describes it as a retrieval-augmented generation design.
The details, if you want them
Who wrote it, and where to read it
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel and Douwe Kiela. The authors were at Facebook AI Research, University College London and New York University. First posted on 22 May 2020 and accepted at NeurIPS 2020.
The two kinds of memory
- Parametric memory: what the model has stored in its own settings. Here that is a text-writing model called BART, with about 400 million parameters.
- Non-parametric memory: the library. Here that is Wikipedia from December 2018, turned into embeddings so it can be searched by meaning.
- The retriever: two copies of BERT. One turns the question into numbers, the other has already done the same for every passage. The closest passages win.
The two versions
- RAG-Sequence picks its passages once and writes the whole answer from them.
- RAG-Token can lean on a different passage for each word it writes.
- Both were trained without anyone marking which passages were the right ones. The system worked that out from whether the final answers were correct.
All the results
- Open questions, exactly right: 44.5 on Natural Questions, 45.5 on WebQuestions and 52.2 on CuratedTrec, the best published at the time on each.
- Writing quiz clues: judges found RAG more specific than the model alone in 37.4% of cases, against 16.8% the other way.
- Checking claims as true or false: 72.5% on a three-way test, within about 4 points of systems built specially for it.
- Answering without the answer in hand: RAG still got 11.8% right when none of the passages it found contained the answer.
- Search matters: a learned search beat plain keyword search on every task except fact-checking.
The limits the authors list
- The library is not perfect. In their words, Wikipedia “will probably never be entirely factual and completely devoid of bias”.
- The searchable index needed about 100 GB of memory, or 36 GB when compressed.
- The same technology could be used to produce misleading content.
Words the paper uses
- Retrieval: searching the library.
- Generation: writing the answer.
- Hallucination: a fluent answer that is not true. See hallucination.
- Provenance: being able to say where a fact came from.
Next paper: LoRA