{"notes":[{"id":"retrieval-augmented-generation/self-rag","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Self-RAG","subtitle":"Three Daily Reflections","author":"Z. Yuan","paperDate":"2023-10-17","publishedOn":"2026-06-21","world":"external/self-rag","paperTitle":"Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection","paperUrl":"https://arxiv.org/abs/2310.11511","bibliography":{"title":"Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection","authors":["Akari Asai","Zeqiu Wu","Yizhong Wang","Avirup Sil","Hannaneh Hajishirzi"],"url":"https://arxiv.org/abs/2310.11511","pdfUrl":"https://arxiv.org/pdf/2310.11511v1","codeUrl":null,"preprint":{"id":"2310.11511","submittedOn":"2023-10-17","url":"https://arxiv.org/abs/2310.11511"},"publication":{"venue":"ICLR","year":2024,"kind":"conference","url":"https://proceedings.iclr.cc/paper_files/paper/2024/hash/25f7be9694d7b32d5cc670927b8091e1-Abstract-Conference.html"},"citation":"@misc{retrievalaugmentedgenerationselfrag2023,\n  title = {{Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection}},\n  author = {Akari Asai and Zeqiu Wu and Yizhong Wang and Avirup Sil and Hannaneh Hajishirzi},\n  year = {2023},\n  eprint = {2310.11511},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2310.11511}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/self-rag.7e4f68374820.bib"},"thumbnail":{"src":"/content/papers/assets/a967fd9c8d63065a0369d6a07be24dc71bb92dab181e5083c3f13ce21c92cabb.png","width":1122,"height":622},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/self-rag.ccfcc6f5169a.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"reflection-tokens","text":"Reflection Tokens","depth":3},{"id":"walking-through-inference","text":"Walking Through Inference","depth":3},{"id":"how-is-the-score-calculated","text":"How Is the Score Calculated?","depth":3},{"id":"how-does-the-model-learn-to-grade-itself","text":"How Does the Model Learn to Grade Itself?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experimental-setup","text":"Experimental Setup","depth":3},{"id":"main-results","text":"Main Results","depth":3},{"id":"which-design-choices-help","text":"Which Design Choices Help?","depth":3},{"id":"other-observations","text":"Other Observations","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"By 2023, the standard RAG recipe was well established: whatever the user asks, retrieve the top $K$ documents, prepend them to the question, and let the language model answer.\n\nThis approach has three often-overlooked problems.\n\nFirst, **some questions need no lookup at all**. Ask the model to write “My Most Memorable Summer,” and fetching five documents from a database will only get in the way.\n\nSecond, **retrieved documents may be irrelevant**. Ask “How did US states get their names?” and the retriever might return “The Most Popular Baby Names in Texas,” because both mention names and states.\n\nThird, **the model may ignore even relevant documents**. Without explicit training to follow the documents, the model may cite them while writing things they never say—or even contradict.\n\nThis paper from the University of Washington, the Allen Institute for AI (AI2), and IBM Research was later published at ICLR 2024; this note follows arXiv v1. The idea is to let the model decide whether to retrieve, evaluate what comes back, and check what it has written.\n\n## Defining the Problem\n\nStart with the paper's first figure. Traditional RAG is on the left; the proposed Self-RAG is on the right:\n\n[![Traditional RAG versus Self-RAG: fixed top-K retrieval and generation on the left; on-demand retrieval, parallel generation, and reflection-token evaluation to choose the best segment on the right](/content/papers/assets/a967fd9c8d63065a0369d6a07be24dc71bb92dab181e5083c3f13ce21c92cabb.png)](/content/papers/assets/a967fd9c8d63065a0369d6a07be24dc71bb92dab181e5083c3f13ce21c92cabb.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v1 PDF**](https://arxiv.org/pdf/2310.11511v1#page=2), CC BY 4.0. Traditional RAG is on the left and Self-RAG on the right. The bottom row is a writing task that needs no retrieval.*\n\nLook at the left side first. The question is “How did US states get their names?” The retriever returns three documents: the first says 11 of the 50 states are named after people; the second discusses popular baby names in Texas; the third says California was named after a fictional island in a Spanish novel.\n\nGiven all three, the model writes that “California was named after Columbus,” marked with a red underline and **Contradictory**: it directly conflicts with the third document. Another sentence says Texas and Utah were named after Native American tribes, marked **No information in passages**: none of the three documents mentions that.\n\nThe bottom row makes the problem even plainer: traditional RAG retrieves three documents even for the summer essay.\n\nThe question the authors want to solve is:\n\n- **Can a single language model judge when to retrieve, whether the retrieved information is useful, and whether its own writing is supported?**\n\n## Solving the Problem\n\n### Reflection Tokens\n\nThe authors add a set of special tokens to the model's vocabulary, called **reflection tokens**. As the model generates text, it also generates these tokens at appropriate points, rather like grading itself while writing.\n\nThere are four types:\n\n| Token | What it examines | Possible values | Meaning |\n| --- | --- | --- | --- |\n| Retrieve | Question and text written so far | yes / no / continue | Whether to retrieve now |\n| IsRel | Question, document | relevant / irrelevant | Whether the document helps answer the question |\n| IsSup | Question, document, output segment | fully / partially / no support | Whether the document supports this output |\n| IsUse | Question, output | 5 / 4 / 3 / 2 / 1 | How useful the answer is overall |\n\n*Compiled from Table 1 of the paper.*\n\nThe last three are collectively called **critique tokens** and perform the evaluation. Retrieve decides whether to look something up.\n\n### Walking Through Inference\n\nReturn to the right side of the figure and follow the same question.\n\nSelf-RAG generates text in **segments**. In the paper's experiments, each segment is a sentence.\n\n**Step 1: Should I retrieve?** The model first writes, “US states' names have various origins.” It then generates a `Retrieve` token, deciding that the next part needs factual support, and calls the retriever to obtain three documents.\n\n**Step 2: Generate in parallel.** Each document is **separately** appended to the question. The model generates three candidate next sentences in parallel and gives each document and candidate critique tokens:\n\n- Document 1: `Relevant`; “11 of the 50 states are named after people”; `Supported`.\n- Document 2: `Irrelevant`; “Texas was named after a Native American tribe”; unsupported.\n- Document 3: `Relevant`; “California's name comes from the 16th-century novel Las Sergas de Esplandián”; `Partially`.\n\n**Step 3: Score and choose the best.** The candidate from Document 1 is relevant and fully supported, so it ranks first. Document 3 comes second and Document 2 last. The model adopts the first candidate and continues, retrieving again when needed.\n\nFor the writing task at the bottom, the model generates `No Retrieval` at the outset and writes directly, like an ordinary language model.\n\n### How Is the Score Calculated?\n\nThe authors express the ranking in Step 3 as a score. For each candidate segment $y_t$ paired with document $d$:\n\n$$\nf(y_t,d)=p(y_t\\mid x,d,y_{<t})+\\mathcal{S}(\\text{Critique})\n$$\n\nThe first term is the probability of generating the sentence. The second is a weighted sum of critique-token scores:\n\n$$\n\\mathcal{S}(\\text{Critique})=\\sum_{G\\in\\{\\text{IsRel},\\,\\text{IsSup},\\,\\text{IsUse}\\}} w^G\\, s^G_t\n$$\n\nHere $s^G_t$ is the probability of the **most desirable value** in that token group: `Relevant` for IsRel, `fully supported` for IsSup, or `5` for IsUse, normalized within the group.\n\nLet's work through some hypothetical numbers. The paper's default weights are $w^{\\text{IsRel}}=1.0$, $w^{\\text{IsSup}}=1.0$, and $w^{\\text{IsUse}}=0.5$. Suppose two candidates have these critique-token probabilities:\n\n| Candidate | IsRel = Relevant | IsSup = fully | IsUse = 5 |\n| --- | ---: | ---: | ---: |\n| Using Document 1 | 0.90 | 0.80 | 0.60 |\n| Using Document 3 | 0.85 | 0.30 | 0.70 |\n\nTheir critique scores are:\n\n$$\n\\begin{aligned}\n\\mathcal{S}_1 &= 1.0\\times0.90+1.0\\times0.80+0.5\\times0.60=2.00\\\\\n\\mathcal{S}_3 &= 1.0\\times0.85+1.0\\times0.30+0.5\\times0.70=1.50\n\\end{aligned}\n$$\n\nIf the sentences' generation probabilities are similar, Document 1's candidate wins.\n\nThe weights can be adjusted at inference time **without retraining**. If your application strongly requires every sentence to have a source, raise $w^{\\text{IsSup}}$ to 2.0. The gap grows from 0.5 to 1.0: 2.80 versus 1.80.\n\nYou can also impose a hard rule: discard any candidate that generates `IsSup = No support`.\n\nFor retrieval, besides letting the model directly generate `Retrieve=Yes`, you can use a threshold: retrieve when its normalized probability exceeds $\\delta$. A lower threshold means more frequent retrieval.\n\nIn the actual search, the authors use beam search of width 2 at the segment level, with greedy decoding for tokens inside each segment.\n\n\n> [!NOTE]\n> \n> The critique-token probabilities above are hypothetical numbers chosen to illustrate the formula, rather than experimental results. The weights 1.0, 1.0, and 0.5 and the beam width of 2 are the paper's default settings.\n\n\n### How Does the Model Learn to Grade Itself?\n\nAt inference time, the model seems to know how to judge and score things on its own. How are those abilities trained?\n\nThe authors use two steps.\n\n**Step 1: Train a critic model $\\mathcal{C}$.** Manually annotating reflection tokens for every sentence would be expensive, so the authors ask GPT-4 to do it. They write an instruction for each token type—for example, “Given an instruction, decide whether finding external documents on the web would help produce a better response”—and add a few demonstrations. They collect 4,000 to 20,000 examples per token type and fine-tune a Llama 2-7B model. The resulting critic agrees with GPT-4 on over 90% of cases in most token categories.\n\n**Step 2: Use the critic to rewrite training data, then train the generator $\\mathcal{M}$.** Run 150,000 instruction–response pairs through the critic sentence by sentence. Insert `Retrieve` and retrieved documents wherever retrieval is needed, IsRel after each document, IsSup after each sentence, and IsUse at the end of the whole answer.\n\nThe processed data looks like this, simplified from Figure 2:\n\n```text\nInput: How did US states get their names?\n\nOutput: [Retrieve]<p>Of the fifty states, eleven are named after an individual person</p>\n        [Relevant] 11 of 50 states' names come from person. [Supported]\n        [Retrieve]<p>LOUISIANA: Named in honor of Louis XIV of France.</p>\n        [Relevant] For instance, Louisiana was named after King Louis XIV, and\n        Georgia was named after King George II. [Partially] [Utility:5]\n```\n\nThen comes ordinary language-model training: predict the next token. The only difference is that retrieved documents between `<p>` and `</p>` are excluded from the loss. They come from the retriever; the model need not learn to write them.\n\nAfter training, the generator produces these tokens itself. The critic is no longer called at inference time.\n\nThe authors also compare this with RLHF. PPO training repeatedly calls a reward model. Self-RAG instead writes critiques into the training data beforehand and uses the standard language-model objective, at much lower cost.\n\n\n> [!TIP]\n> \n> **“Self-RAG” Tutorials and This Paper Can Refer to Different Implementations**\n> \n> Many online tutorials, such as LangGraph's archived [**Self-RAG example**](https://github.com/langchain-ai/langgraph/blob/fbcb8a911b626b4373fd4aff9624124e9532987f/examples/rag/langgraph_self_rag.ipynb), borrow the paper's decision flow: is the document relevant, is the answer grounded, and does it answer the question?\n> \n> Their implementation differs. These examples make a separate LLM call and prompt it for yes or no, then use that result to retrieve again, rewrite the question, or regenerate the answer. The paper **trains these judgments into tokens the generator itself emits**, within the same generation, with probabilities that can be weighted for ranking.\n> \n> Both are usable. When you encounter “Self-RAG,” check which one is meant.\n\n\n## Discussion\n\n### Experimental Setup\n\nEvery evaluation is zero-shot: provide task instructions without demonstrations.\n\n- **Closed-set tasks**: PubHealth for public-health fact verification and ARC-Challenge for science questions, scored by accuracy.\n- **Short answers**: A long-tail PopQA subset of 1,399 questions about obscure entities whose Wikipedia pages receive fewer than 100 monthly views, plus 11,313 questions from TriviaQA-unfiltered. An answer is correct if a reference answer appears in the output.\n- **Long-form generation**: FactScore evaluates factuality in biographies. ASQA long-answer QA additionally evaluates citation precision and recall.\n\nThe generators are Llama 2-7B and 13B. The retriever is an off-the-shelf Contriever-MS MARCO, returning the top five documents by default. Biographies and open-domain QA additionally use the top five web search results.\n\n### Main Results\n\nTable 2 compares many models. Here are some particularly informative rows:\n\n| Model | PopQA | PubHealth | Biography FactScore | ASQA citation precision |\n| --- | ---: | ---: | ---: | ---: |\n| ChatGPT | 29.3 | 70.1 | 71.8 | — |\n| ChatGPT + retrieval | 50.8 | 54.7 | — | 65.1 |\n| Llama2-chat 13B + retrieval | 51.8 | 52.1 | 79.9 | 19.8 |\n| Llama2-FT 7B + retrieval | 48.7 | 64.3 | 78.2 | 5.0 |\n| **Self-RAG 7B** | 54.9 | 72.4 | 81.2 | 66.9 |\n| **Self-RAG 13B** | 55.8 | 74.5 | 80.2 | 70.3 |\n\n*Compiled from Table 2 of the paper.*\n\nSeveral things stand out.\n\n**Retrieval can hurt.** Adding retrieval to ChatGPT raises PopQA from 29.3 to 50.8: obscure facts really do need looking up. But PubHealth falls from 70.1 to 54.7. That is the first problem from the opening: forcing retrieval where it is unnecessary can be harmful.\n\n**Citation precision shows the largest gap.** Ordinary RAG baselines mostly sit between single digits and about 20%; Self-RAG 13B reaches 70.3, above even ChatGPT with retrieval at 65.1. Citation precision asks whether a sentence citing a document is fully supported by that document.\n\n**The training data alone cannot explain the gain.** Llama2-FT 7B uses the exact same 150,000 instruction examples as Self-RAG, but without reflection tokens or retrieved documents; retrieval is added at test time. It trails Self-RAG 7B on every metric except ASQA str-em and ROUGE, where it is slightly higher: 31.0 versus 30.0 and 35.8 versus 35.7.\n\nSelf-RAG does not win everywhere. ChatGPT scores 74.3 on TriviaQA and 75.3 on ARC-Challenge, ahead of Self-RAG 13B's 69.3 and 73.1. ChatGPT with retrieval also has higher ASQA correctness (str-em) and citation recall.\n\n### Which Design Choices Help?\n\n[![Self-RAG analysis: (a) training and inference ablations, (b) effects of IsSup weights on citation precision and MAUVE, and (c) effects of the retrieval threshold on accuracy and retrieval frequency](/content/papers/assets/ed98a0625c59ca4eaa3a682d3fddfda9963f66ff78e56b8a320597c846caf562.png)](/content/papers/assets/ed98a0625c59ca4eaa3a682d3fddfda9963f66ff78e56b8a320597c846caf562.png)\n\n*Figure 3 from the paper, excerpted from [**page 9 of the v1 PDF**](https://arxiv.org/pdf/2310.11511v1#page=9), CC BY 4.0. (a) Ablations; (b) inference-time weight adjustment; (c) retrieval-threshold adjustment.*\n\n**(a) Ablations.** To save resources, these experiments use a 7B model trained on 50,000 examples, tested on PopQA (PQA), PubHealth (Med), and ASQA (AS).\n\n- Removing the critic during training (No Critic: attach one retrieved document to every example without reflection tokens) drops ASQA from 32.1 to 18.1.\n- Always using only the top-ranked document at inference (Retrieve top1, the closest to traditional RAG) drops PopQA from 45.5 to 41.8 and ASQA to 28.6.\n- Disabling retrieval entirely (No retrieval) leaves PopQA at 24.7, while PubHealth barely changes at 73.0.\n\nThe last result echoes the main findings: PopQA needs lookup; PubHealth mostly does not.\n\n**(b) Adjusting IsSup weights.** The horizontal axis is $w^{\\text{IsSup}}$. Increasing it raises citation precision but lowers MAUVE, a fluency metric. The authors explain that longer, more fluent sentences are more likely to include content not fully supported by their citations. This is the promised ability to adjust model behavior without retraining.\n\n**(c) Adjusting the retrieval threshold.** A higher threshold lowers retrieval frequency. Retrieving less barely affects PubHealth accuracy, but substantially hurts PopQA.\n\n### Other Observations\n\n- **More training data helps.** Increasing from 5,000 to 150,000 examples produces particularly clear gains on PopQA and ASQA. The authors think further scaling could help.\n- **A small human evaluation.** The authors manually inspect 50 examples each from PopQA and the biography outputs. Predicted IsRel agrees with human judgments in 95.0% and 90.0% of cases, respectively; IsSup agreement is 90.0% and 85.0%.\n- **Inference cost.** Each retrieval processes multiple documents in parallel, generates multiple candidates, and runs segment-level beam search. The authors use vLLM for acceleration, but report no latency or cost comparison with traditional RAG. Actual deployment requires its own measurements.\n\n## Conclusion\n\nSelf-RAG hands several hard-coded RAG decisions back to the model:\n\n1. **Whether to retrieve**: `Retrieve` enables on-demand retrieval; writing tasks can skip it.\n2. **Which document helps**: `IsRel` evaluates each retrieved document.\n3. **Whether the writing is grounded**: `IsSup` checks whether each sentence is supported.\n4. **Whether the answer is useful overall**: `IsUse` evaluates overall answer quality.\n\nGPT-4 annotates these tokens; the annotations are distilled into a critic, written into training data, and eventually learned by the generator itself. At inference, token probabilities form scores, and weights adjust the model's trade-offs without retraining.\n\nIn the paper's experiments, Self-RAG 7B and 13B outperform the ChatGPT and Llama2-chat comparisons with reported scores on PopQA, PubHealth, biographies, and ASQA citation precision. ChatGPT remains ahead on tasks such as TriviaQA and ARC-Challenge.\n\nAlways retrieving the top $K$ documents used to be RAG's default. This paper reminds us that deciding whether to look something up—and whether to use what comes back—is itself part of the task.\n","imageSizes":{"/content/papers/assets/a967fd9c8d63065a0369d6a07be24dc71bb92dab181e5083c3f13ce21c92cabb.png":{"width":1122,"height":622},"/content/papers/assets/ed98a0625c59ca4eaa3a682d3fddfda9963f66ff78e56b8a320597c846caf562.png":{"width":1122,"height":447}}}