{"notes":[{"id":"retrieval-augmented-generation/rag","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"RAG","subtitle":"Why Are We Calling It a Rag?","author":"Z. Yuan","paperDate":"2020-05-22","publishedOn":"2026-05-17","world":"external/rag","paperTitle":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","paperUrl":"https://arxiv.org/abs/2005.11401","bibliography":{"title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","authors":["Patrick Lewis","Ethan Perez","Aleksandra Piktus","Fabio Petroni","Vladimir Karpukhin","Naman Goyal","Heinrich Küttler","Mike Lewis","Wen-tau Yih","Tim Rocktäschel","Sebastian Riedel","Douwe Kiela"],"url":"https://arxiv.org/abs/2005.11401","pdfUrl":"https://arxiv.org/pdf/2005.11401v4","codeUrl":null,"preprint":{"id":"2005.11401","submittedOn":"2020-05-22","url":"https://arxiv.org/abs/2005.11401"},"publication":{"venue":"NeurIPS","year":2020,"kind":"conference","url":"https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html"},"citation":"@misc{retrievalaugmentedgenerationrag2020,\n  title = {{Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks}},\n  author = {Patrick Lewis and Ethan Perez and Aleksandra Piktus and Fabio Petroni and Vladimir Karpukhin and Naman Goyal and Heinrich Küttler and Mike Lewis and Wen-tau Yih and Tim Rocktäschel and Sebastian Riedel and Douwe Kiela},\n  year = {2020},\n  eprint = {2005.11401},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2005.11401}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/rag.85ab2502cd0c.bib"},"thumbnail":{"src":"/content/papers/assets/05c7bcd1d2c3f13c0726dfa7180e85f043b7cd64a0c61fa595b954d979a6550e.png","width":1122,"height":358},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/rag.2e04c38f4e0c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"architecture","text":"Architecture","depth":3},{"id":"external-memory-21-million-wikipedia-passages","text":"External Memory: 21 Million Wikipedia Passages","depth":3},{"id":"the-document-is-a-random-variable","text":"The Document Is a Random Variable","depth":3},{"id":"how-does-the-retriever-learn-without-document-labels","text":"How Does the Retriever Learn Without Document Labels?","depth":3},{"id":"decoding-at-inference-time","text":"Decoding at Inference Time","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"open-domain-question-answering","text":"Open-Domain Question Answering","depth":3},{"id":"jeopardy-clues-combining-two-documents","text":"Jeopardy Clues: Combining Two Documents","depth":3},{"id":"is-the-learned-retriever-better","text":"Is the Learned Retriever Better?","depth":3},{"id":"replace-the-index-update-the-knowledge","text":"Replace the Index, Update the Knowledge","depth":3},{"id":"where-things-were-less-smooth","text":"Where Things Were Less Smooth","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"These days, whenever someone talks about getting a language model to read company documents, “RAG” is almost bound to come up.\n\nThe name comes from this 2020 paper.\n\nOf course, a rag is also a piece of old cloth. A technique that would eventually sweep through the AI industry ended up sharing its name with something used to wipe the floor. You have to laugh.\n\nIn an [**interview with NVIDIA**](https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/), first author Patrick Lewis later apologized for the unfortunate acronym. Had they known how widely it would spread, they would have put more thought into the name. They had meant to come up with something better, but nobody did while writing the paper.\n\nSo the name stuck.\n\nIf you first encountered RAG after ChatGPT, though, this paper may surprise you. Today's usual recipe is to retrieve a few passages, paste them into a prompt, and ask a large language model to answer. Here, RAG uses a 400-million-parameter BART, trains the retriever and generator together, and treats the retrieved document as a **random variable**.\n\nLet us go back to what the name originally referred to.\n\n## Defining the Problem\n\nAround 2020, pretrained language models were already known to store substantial factual knowledge in their parameters. Models such as T5 could answer some questions directly, without any reference material.\n\nThis is called **closed-book** question answering. Like a closed-book exam, all the knowledge has to be in your head.\n\nThat creates several problems:\n\n1. **Knowledge is hard to update.** The world changes, but the knowledge in the parameters stays put until the model is trained again.\n2. **Sources are unavailable.** Even when the answer is correct, you do not know where the model learned it.\n3. **The model can invent things with a straight face.** What we now routinely call hallucination.\n\nAnother approach is **open-book**: let the model bring external material to the exam. Before this paper, REALM and ORQA had already connected differentiable retrievers to masked language models, but for **extractive question answering**. The answer had to be a span in the source text; the model's job was to locate it.\n\nThe authors wanted something broader:\n\n- **Could a general sequence-to-sequence generator carry a searchable external memory and use the same approach to fine-tune for different tasks?**\n\nOne condition makes this harder: training data contains questions and answers, but nobody has labeled which document to retrieve.\n\nHow is the retriever supposed to learn?\n\n## Solving the Problem\n\n### Architecture\n\n[![RAG overview: the Query Encoder encodes the input as q(x), MIPS retrieves top-K documents from the Document Index, and the Generator produces an answer while marginalizing over documents](/content/papers/assets/05c7bcd1d2c3f13c0726dfa7180e85f043b7cd64a0c61fa595b954d979a6550e.png)](/content/papers/assets/05c7bcd1d2c3f13c0726dfa7180e85f043b7cd64a0c61fa595b954d979a6550e.png)\n\n*Figure 1 of the paper, from [**page 2 of the v4 PDF**](https://arxiv.org/pdf/2005.11401v4#page=2). Inputs for three tasks appear on the left, with their outputs on the right.*\n\nRead the diagram from left to right.\n\nAt the far left is the input $x$: a question (“define middle ear”), a statement to fact-check, or an answer term from which to generate a Jeopardy clue (“The Divine Comedy”).\n\nThe green box is the **retriever**, $p_\\eta$. The Query Encoder maps the input to a vector $q(x)$; the Document Index stores a precomputed vector $d(z)$ for each document. MIPS selects the most relevant documents, $z_1,\\dots,z_4$.\n\nThe blue box is the **generator**, $p_\\theta$. It concatenates the input with each document separately and generates an answer. “Marginalize” then combines the document-conditioned results into a prediction.\n\nThe dashed arrow at the top reads “End-to-End Backprop through **q** and $p_\\theta$.” During training, gradients flow back to the Query Encoder and generator. Notice that the Document Index is not mentioned. We will return to that.\n\nThe authors describe the two components as two kinds of memory:\n\n- **Parametric memory**: the generator's parameters—the knowledge BART has learned.\n- **Non-parametric memory**: the external document index, which can be replaced directly.\n\n### External Memory: 21 Million Wikipedia Passages\n\nThere is one external knowledge source: the December 2018 English Wikipedia dump.\n\nArticles are split into non-overlapping passages of 100 words, yielding about 21 million “documents.” A document encoder produces their vectors, which are stored in an index built with FAISS.\n\nThe retriever is [**DPR**](https://arxiv.org/abs/2004.04906), the Dense Passage Retriever. It uses two BERT-base encoders, one for questions and one for documents. Relevance is the inner product of their vectors:\n\n$$\np_\\eta(z\\mid x)\\propto \\exp\\!\\left(d(z)^\\top q(x)\\right),\\quad d(z)=\\text{BERT}_d(z),\\ q(x)=\\text{BERT}_q(x)\n$$\n\nFinding the $K$ largest inner products among 21 million vectors is a **MIPS**, or Maximum Inner Product Search, problem.\n\n\n> [!TIP]\n> \n> **Do we have to score all 21 million vectors every time?**\n> \n> No. The authors use FAISS with HNSW, Hierarchical Navigable Small World, for approximate search. Vectors are organized into a multilayer neighborhood graph. A query moves through the graph toward its target rather than computing an inner product with every vector. The trade-off is that the returned documents may not be the exact top $K$, in exchange for sublinear query time.\n> \n> The appendix also notes that the index contains 15.3 billion values, which can be stored at 8-bit precision to reduce memory and disk use.\n\n\nThe generator is BART-large, with about 400 million parameters. Concatenate input $x$ and document $z$, then let it generate the answer. That part is straightforward.\n\n### The Document Is a Random Variable\n\nNow we reach the paper's central idea.\n\nAs noted above, training data does not identify the right document. The authors' solution is to **leave that choice unresolved**. Treat the retrieved document $z$ as a latent variable and take a weighted average over the possible documents.\n\nThe retriever assigns each document a probability $p_\\eta(z\\mid x)$. Conditioned on that document, the generator assigns the answer a probability $p_\\theta(y\\mid x,z)$. Multiply the two and sum across documents to obtain the answer probability.\n\nThere are two ways to arrange that summation.\n\n**RAG-Sequence** ties the entire answer to one document. Compute the probability of the whole sequence under each document, then weight across documents:\n\n$$\np_{\\text{RAG-Sequence}}(y\\mid x)\\approx\\sum_{z\\in\\text{top-}k}p_\\eta(z\\mid x)\\prod_{i}^{N}p_\\theta(y_i\\mid x,z,y_{1:i-1})\n$$\n\n**RAG-Token** allows each token to draw on a different document. At each step, average across documents to obtain that token's probability, then multiply over tokens:\n\n$$\np_{\\text{RAG-Token}}(y\\mid x)\\approx\\prod_{i}^{N}\\sum_{z\\in\\text{top-}k}p_\\eta(z\\mid x)\\,p_\\theta(y_i\\mid x,z,y_{1:i-1})\n$$\n\nOnly the order of $\\sum$ and $\\prod$ changes. What difference does that make?\n\nLet us try some small numbers.\n\nSuppose the retriever finds two documents, assigning doc1 probability 0.6 and doc2 probability 0.4. The answer has two tokens:\n\n- Under doc1, the generator assigns the first token probability 0.9 and the second 0.2.\n- Under doc2, the pattern reverses: 0.3 for the first token and 0.8 for the second.\n\nIn other words, doc1 supports the first part of the answer, while doc2 supports the second.\n\n```python\n# Retriever probabilities for the two documents\np_z = {\"doc1\": 0.6, \"doc2\": 0.4}\n\n# Generator probabilities for the two answer tokens under each document\np_y = {\n    \"doc1\": [0.9, 0.2],  # Strong support for token 1, weaker support for token 2\n    \"doc2\": [0.3, 0.8],  # The reverse\n}\n\n# RAG-Sequence: one document for the whole answer, then average over documents\nseq = sum(p_z[z] * p_y[z][0] * p_y[z][1] for z in p_z)\n\n# RAG-Token: average over documents separately for each token, then multiply\ntok = 1.0\nfor i in range(2):\n    tok *= sum(p_z[z] * p_y[z][i] for z in p_z)\n\nprint(f\"RAG-Sequence: {seq:.4f}\")\nprint(f\"RAG-Token:    {tok:.4f}\")\n\n# RAG-Sequence: 0.2040\n# RAG-Token:    0.2904\n```\n\nRAG-Sequence needs one document to support the whole answer. Doc1 supports the first half but struggles with the second; doc2 has the opposite problem. Both paths are penalized.\n\nRAG-Token draws mainly on doc1 for the first token and doc2 for the second, giving this assembled answer a higher probability.\n\nThis also explains why RAG-Token performs better on Jeopardy clue generation in the experiments. We will see a concrete example in the discussion.\n\n\n> [!NOTE]\n> \n> **What about classification?**\n> \n> Treat a class label as a one-token answer—for example, “supports,” “refutes,” or “not enough information” in fact-checking. With a single token, changing the order of $\\sum$ and $\\prod$ has no effect, so the two RAG formulations coincide.\n\n\n### How Does the Retriever Learn Without Document Labels?\n\nThe objective is simple: maximize the marginal probability of the correct answer, or minimize $-\\log p(y\\mid x)$.\n\nLook back at the equations. Since $p_\\eta(z\\mid x)$ appears inside the sum, gradients reach it. If a document helps the generator produce the correct answer, increasing that document's probability raises the overall marginal probability. Training teaches the retriever to score such documents more highly.\n\nNobody tells it which document is right. It learns from which documents help.\n\nHowever, the authors update only the **question encoder**, $\\text{BERT}_q$, and BART. **The document encoder and index remain frozen.**\n\nThe reason is practical: change the document encoder and all 21 million document vectors must be recomputed and the index rebuilt. REALM periodically rebuilds its index during pretraining; the authors found that their results were already good without this step.\n\n\n> [!TIP]\n> \n> **A common misunderstanding about “end-to-end retriever training”**\n> \n> Some explanations say RAG trains its retriever end to end. More precisely, training updates the question side. The document side remains at its DPR initialization. DPR itself was already trained with retrieval labels from Natural Questions and TriviaQA, a point the authors explicitly mention when discussing results.\n\n\n### Decoding at Inference Time\n\nRAG-Token produces a next-token probability at each step, so standard beam search applies directly.\n\nRAG-Sequence is trickier: its sequence probability cannot be decomposed into individual next-token probabilities in the same way. The authors run beam search separately for each document, collect all candidate answers, then compute each candidate's probability under the other documents and take the weighted sum. They call this “Thorough Decoding.”\n\nWith many candidates, those additional forward passes become expensive. An approximation, “Fast Decoding,” assigns zero probability under a document to any candidate absent from that document's beam, avoiding the extra computation.\n\n## Discussion\n\n### Open-Domain Question Answering\n\nThe authors evaluate on four open-domain QA datasets: Natural Questions (NQ), TriviaQA (TQA), WebQuestions (WQ), and CuratedTrec (CT). The metric is Exact Match, the proportion of answers matching the reference exactly.\n\n[![Table 1: test scores for closed-book T5, open-book REALM and DPR, RAG-Token, and RAG-Sequence on NQ, TQA, WQ, and CT](/content/papers/assets/e4210dd8d6a0578d12df7e28472ad4aec7137a2aa87c895aad1e7044d0bfcdd6.png)](/content/papers/assets/e4210dd8d6a0578d12df7e28472ad4aec7137a2aa87c895aad1e7044d0bfcdd6.png)\n\n*Table 1 of the paper, from [**page 6 of the v4 PDF**](https://arxiv.org/pdf/2005.11401v4#page=6). The left TQA column uses the standard open-domain test set; the right uses TQA-Wiki.*\n\nThe table has three sections: closed-book T5 at the top, open-book REALM and DPR in the middle, and RAG at the bottom.\n\nOn NQ, RAG-Sequence scores 44.5, compared with 34.5 for closed-book T5-11B.\n\nParameter counts matter here. T5-11B has 11 billion trainable parameters. RAG has about 626 million in total: two BERT-base encoders of roughly 110 million each, with the document encoder actually frozen, plus BART's approximately 400 million. The appendix notes that T5-large, the closest in size at 770 million parameters, scores only 28.9 on NQ.\n\nAnother comparison is DPR's QA system, which additionally uses a cross-encoder reranker and an extractive reader. RAG uses neither and still scores 3 points higher on NQ.\n\nGeneration also has an advantage extraction cannot provide: when a document contains clues but not the literal answer, the model may still generate the correct response. Among NQ questions whose answers were absent from every retrieved document, RAG answered 11.8% correctly. An extractive model necessarily scores 0% in that situation.\n\n### Jeopardy Clues: Combining Two Documents\n\nJeopardy is an American quiz show with an unusual format: given an answer, produce its clue. For “The World Cup,” a clue might read, “In 1986, Mexico became the first country to host this international event twice.”\n\nThese clues often combine two facts, making them a useful test of RAG-Token's ability to assemble information. The figure below shows which document the model considers helpful at each generated token for the input “Hemingway,” with five documents retrieved:\n\n[![Posterior probability heatmap over five documents for each token generated by RAG-Token when producing a Jeopardy clue for Hemingway](/content/papers/assets/bc3d0fd74e0299ec9fc8eae15768959cef1e43472643f9449c1f7002164a42db.png)](/content/papers/assets/bc3d0fd74e0299ec9fc8eae15768959cef1e43472643f9449c1f7002164a42db.png)\n\n*Figure 2 of the paper, from [**page 7 of the v4 PDF**](https://arxiv.org/pdf/2005.11401v4#page=7). Generated tokens run horizontally and retrieved documents vertically; darker colors indicate higher posterior probabilities.*\n\nTwo documents are shown on the left. Document 1 mentions *A Farewell to Arms*, and Document 2 mentions *The Sun Also Rises*.\n\nLook at the heatmap on the right. When the model generates “Sun,” Doc 2's row becomes dark. When it generates “A” to begin “A Farewell to Arms,” Doc 1 takes over.\n\nThis is the situation we just calculated: different documents support different parts of one sentence.\n\nThere is another interesting detail. After the first token of a book title, the heatmap fades. The authors interpret this as BART knowing how to finish the title itself. They fed just “\\\"The Sun” to BART without retrieval, and it completed the title. In their account, the two memories cooperate: a retrieved document points the model in the right direction, and parametric knowledge finishes the phrase.\n\nHuman evaluation compared factual correctness in 452 pairs of BART and RAG-Token clues. BART was judged more factual in 7.1% of pairs, versus 42.7% for RAG, according to Table 4.\n\n### Is the Learned Retriever Better?\n\nThe authors test two ablations: freeze the retriever instead of training it, or replace it with BM25 keyword retrieval.\n\nFor RAG-Sequence on the NQ development set, Table 6 gives:\n\n1. **BM25 replacement**: 31.8.\n2. **Frozen retriever**: 41.2.\n3. **Joint training**: 44.0.\n\nLearned retrieval makes a substantial difference for QA.\n\nFEVER fact-checking is an exception. BM25 reaches 75.1 three-way classification accuracy, slightly above learned retrieval's 74.5. The authors suggest that FEVER claims often concern specific entities: matching a name is a good way to find the material, making keyword retrieval well suited to the task.\n\n### Replace the Index, Update the Knowledge\n\nThis experiment makes the value of external memory particularly clear.\n\nThe authors build another index from December 2016 Wikipedia, then collect 82 world-leader positions whose occupants changed between 2016 and 2018. They ask “Who is {position}?,” for example, “Who is the President of Peru?”\n\n| Index | Questions about 2016 leaders | Questions about 2018 leaders |\n| --- | ---: | ---: |\n| 2016 index | 70% | 4% |\n| 2018 index | 12% | 68% |\n\n*Compiled from §4.5 of the paper.*\n\nMatch the index year to the questions and roughly seven in ten answers are correct. Mismatch them and accuracy drops to single digits or a little above ten percent.\n\nNo model parameters change. Replacing the external memory changes the answers—something a closed-book model cannot achieve this way.\n\n### Where Things Were Less Smooth\n\n1. **More retrieval is not always better.** On NQ, RAG-Sequence improves as more documents are retrieved at test time; RAG-Token peaks at around 10.\n2. **Retrieval can collapse.** The appendix describes preliminary story-generation experiments where the retriever returned the same documents regardless of input. The generator learned to ignore them, reducing the system to ordinary BART. The authors suggest that such tasks may not clearly require factual knowledge, or that long answers provide less useful gradients to the retriever.\n3. **Wikipedia cannot answer every question.** MS-MARCO includes questions such as “What is the weather in Volcano, California right now?” For questions less likely to benefit from retrieval, the authors observed the model consistently retrieving a particular set of documents.\n\n## Conclusion\n\nThis paper connects a pretrained generator to a searchable external memory, then fine-tunes the same formulation for question answering, clue generation, and fact-checking.\n\nIts most persuasive results are these:\n\n1. **A smaller model with external memory beats a much larger closed-book model.** On NQ, RAG-Sequence's 626 million parameters yield 44.5, compared with 34.5 for the 11-billion-parameter T5-11B.\n2. **Document-selection labels are unnecessary.** Marginalizing over documents as latent variables lets the retriever learn what to retrieve from the answer probabilities.\n3. **Replacing the index updates knowledge.** Switching the 2016 and 2018 indexes changes answers about leaders without changing model parameters.\n\nLooking at today's RAG, two ideas have survived: retrieve before handing documents to the generator, and keep external memory replaceable. The other half—joint retriever-generator training and document marginalization—is much less common in later LLM applications built around prompt concatenation. Models have become expensive to fine-tune, and retrievers are often replaced with ready-made embedding models or search engines.\n\nThat is what developed afterward, rather than a claim made by this paper.\n\nThe name, at least, never changed.\n","imageSizes":{"/content/papers/assets/05c7bcd1d2c3f13c0726dfa7180e85f043b7cd64a0c61fa595b954d979a6550e.png":{"width":1122,"height":358},"/content/papers/assets/e4210dd8d6a0578d12df7e28472ad4aec7137a2aa87c895aad1e7044d0bfcdd6.png":{"width":561,"height":375},"/content/papers/assets/bc3d0fd74e0299ec9fc8eae15768959cef1e43472643f9449c1f7002164a42db.png":{"width":1122,"height":210}}}