{"notes":[{"id":"retrieval-augmented-generation/self-route","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Self-Route","subtitle":"Knowing When You Don't Know","author":"Z. Yuan","paperDate":"2024-07-23","publishedOn":"2026-08-12","paperTitle":"Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach","paperUrl":"https://aclanthology.org/2024.emnlp-industry.66/","bibliography":{"title":"Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach","authors":["Zhuowan Li","Cheng Li","Mingyang Zhang","Qiaozhu Mei","Michael Bendersky"],"url":"https://aclanthology.org/2024.emnlp-industry.66/","pdfUrl":"https://aclanthology.org/2024.emnlp-industry.66.pdf","codeUrl":null,"preprint":{"id":"2407.16833","submittedOn":"2024-07-23","url":"https://arxiv.org/abs/2407.16833"},"publication":{"venue":"EMNLP Industry Track","year":2024,"kind":"conference","url":"https://aclanthology.org/2024.emnlp-industry.66/"},"citation":"@misc{retrievalaugmentedgenerationselfroute2024,\n  title = {{Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach}},\n  author = {Zhuowan Li and Cheng Li and Mingyang Zhang and Qiaozhu Mei and Michael Bendersky},\n  year = {2024},\n  eprint = {2407.16833},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2407.16833}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/self-route.9d3f24607371.bib"},"thumbnail":{"src":"/content/papers/assets/977e711ec46055e2207ae7f021de5d8f1140e73a8d3dadcc2c3daf7a8d146a43.png","width":639,"height":420},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/self-route.7a4683811f7c.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"comparison-setup","text":"Comparison Setup","depth":3},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"first-compare-them-lc-wins","text":"First, Compare Them: LC Wins","depth":3},{"id":"an-interesting-finding-they-often-give-the-same-answer","text":"An Interesting Finding: They Often Give the Same Answer","depth":3},{"id":"self-route-let-the-model-say-i-dont-know","text":"Self-Route: Let the Model Say “I Don't Know”","depth":3},{"id":"how-much-does-it-save-lets-calculate","text":"How Much Does It Save? Let's Calculate","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"results","text":"Results","depth":3},{"id":"how-many-chunks-should-we-retrieve","text":"How Many Chunks Should We Retrieve?","depth":3},{"id":"why-does-rag-fail","text":"Why Does RAG Fail?","depth":3},{"id":"the-trap-in-synthetic-data","text":"The Trap in Synthetic Data","depth":3},{"id":"later-criticism","text":"Later Criticism","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"By mid-2024, long-context models were advancing quickly.\n\nGemini 1.5 Pro could take a million tokens, while GPT-4o supported 128K. A document hundreds of pages long could fit directly into the prompt.\n\nDid we still need RAG?\n\nMany teams were asking that question. Giving the model the entire document skips retrieval and sounds simple and reliable. But sending a hundred thousand tokens for every question also produces a substantial bill.\n\nThis paper from Google DeepMind and the University of Michigan, published in the EMNLP 2024 Industry Track, does two things: compare RAG with long context on the same datasets, then use the findings to design a hybrid.\n\n## Defining the Problem\n\nAn earlier comparison by Xu et al. (2023) concluded that long-context models still lagged behind RAG. These authors argue that models have improved considerably over the past year, so that conclusion deserves another look.\n\nTheir first question is:\n\n- **With the latest models, which is more accurate: RAG or long context (LC)?**\n\nIf LC is more accurate, a second question follows:\n\n- **LC costs more. Can we use it only when needed?**\n\n### Comparison Setup\n\nThe authors select nine English datasets from LongBench and ∞Bench, requiring real text and explicit questions:\n\n- **LongBench**: NarrativeQA (novels and scripts), Qasper (papers), MultiFieldQA, HotpotQA, 2WikiMultihopQA, MuSiQue (the last three are Wikipedia multi-hop QA), and QMSum (meeting transcripts).\n- **∞Bench**: En.QA and En.MC, averaging about 147,000 words—much longer than the other datasets.\n\nOpen-ended QA uses F1, multiple-choice questions use accuracy, and summarization uses ROUGE.\n\nThe three models are Gemini-1.5-Pro (one million tokens), GPT-4O (128K, `gpt-4o-2024-05-13`), and GPT-3.5-Turbo (16K, `gpt-3.5-turbo-0125`).\n\nThe RAG setup is standard: split long texts into 300-word chunks, use one of two dense retrievers, Contriever or Dragon, select the top five chunks by cosine similarity to the query, and put them into the prompt in score order.\n\n\n> [!TIP]\n> \n> **Has the Model Already Memorized the Answer?**\n> \n> Many of these datasets draw on Wikipedia, which commercial models may have seen during training. The authors sometimes find answers with exactly the same wording as the reference—“meticulously,” for example—even though that word never appears in the provided document.\n> \n> Both RAG and LC prompts therefore include “based only on the provided passage.” The appendix verifies that this instruction lowers Gemini-1.5-Pro's average LC score from 50.57 to 45.53, showing that it restricts the use of memorized knowledge. The authors acknowledge that evaluation-data leakage remains an open problem.\n\n\n## Solving the Problem\n\n### First, Compare Them: LC Wins\n\n[![Table 1 of the paper: LC, RAG, and Self-Route scores across nine datasets for Gemini-1.5-Pro, GPT-4O, and GPT-3.5-Turbo, with answerable and token ratios](/content/papers/assets/bdedfb9a9df07b3ecad6c4f8fee397d8b051787576334f930ad25ea74f7be78e.png)](/content/papers/assets/bdedfb9a9df07b3ecad6c4f8fee397d8b051787576334f930ad25ea74f7be78e.png)\n\n*Table 1 from the paper, excerpted from [**page 4 of the ACL Anthology PDF**](https://aclanthology.org/2024.emnlp-industry.66.pdf#page=4), CC BY 4.0. Retrieval uses Contriever. Bold marks the highest score for each model on each dataset.*\n\nThere is a lot in this table. Start with the first two rows for each model: `*-1 LC` and `*-2 RAG`. The leftmost Avg column averages the nine datasets.\n\n| Model | LC | RAG | Gap |\n| --- | ---: | ---: | ---: |\n| Gemini-1.5-Pro | 49.70 | 37.33 | 12.37 |\n| GPT-4O | 48.67 | 32.60 | 16.07 |\n| GPT-3.5-Turbo | 32.07 | 30.33 | 1.74 |\n\n*Calculated from the average scores in Table 1 of the paper.*\n\nLC is better for all three models, with larger gaps for newer models.\n\nThe exceptions are GPT-3.5-Turbo on En.QA and En.MC: RAG scores 15.39 and 43.67, while LC gets only 14.73 and 34.50. These datasets average about 147,000 words, far beyond GPT-3.5-Turbo's 16K-token window. The full documents simply do not fit. The authors identify this as a clear use case for RAG: inputs far longer than the context window.\n\n\n> [!NOTE]\n> \n> **A Note on the Paper's Numbers**\n> \n> Several numbers in the prose differ from Table 1. For example, the prose says LC outperforms RAG on average by 7.6%, 13.1%, and 3.6%; gives Gemini's answerable ratio as 81.74%; and lists GPT-4O's Self-Route and LC scores as 46.83 and 47.04. Table 1 instead gives the gaps shown above, 76.78%, and 48.89 versus 48.67. This note consistently uses Table 1.\n\n\n### An Interesting Finding: They Often Give the Same Answer\n\nLooking only at average scores might suggest “use LC if the budget allows.”\n\nBut the authors then compare answers question by question and draw this figure:\n\n[![Distribution of per-question RAG-minus-LC score differences, with most of the mass at zero](/content/papers/assets/977e711ec46055e2207ae7f021de5d8f1140e73a8d3dadcc2c3daf7a8d146a43.png)](/content/papers/assets/977e711ec46055e2207ae7f021de5d8f1140e73a8d3dadcc2c3daf7a8d146a43.png)\n\n*Figure 2 from the paper, excerpted from [**page 3 of the ACL Anthology PDF**](https://aclanthology.org/2024.emnlp-industry.66.pdf#page=3), CC BY 4.0. The horizontal axis is RAG score minus LC score on the same question; colors indicate their average score.*\n\nThe horizontal axis subtracts LC's score from RAG's for each question, with scores scaled to 0–100. There is a very tall bar at zero: **on 63% of questions, the two answers are identical**. On 70%, the score difference is within ten points.\n\nNow look at the colors. The bottom of that bar is blue: both answers are correct. Its upper portion is red: both are wrong.\n\nRAG and LC frequently succeed together, and frequently fail together too.\n\nIf RAG performs as well as LC on most questions, we only need to identify the ones it cannot handle and send those to LC.\n\n### Self-Route: Let the Model Say “I Don't Know”\n\nHow can we tell which questions RAG cannot handle?\n\nThe authors' answer is direct: **ask the model itself.**\n\nSelf-Route has two steps:\n\n1. **RAG with routing.** Give the model the question and retrieved chunks, adding an instruction: “If the question cannot be answered from the provided text, write unanswerable.” If it answers, use that answer.\n2. **Long-context prediction.** Only questions marked unanswerable are answered again with the full document using LC.\n\nIn code, it looks roughly like this:\n\n```python\ndef self_route(question, document, k=5):\n    chunks = retrieve(question, document, k=k)       # 300 words per chunk; retrieve top k\n    answer = llm(question, context=chunks, allow_unanswerable=True)\n    if answer != \"unanswerable\":\n        return answer                                # Most questions end here\n    return llm(question, context=document)           # Only some need the full document\n```\n\nNo additional classifier is trained. Routing relies entirely on the model's judgment after reading the chunks. The authors assume it can judge sufficiently accurately whether those chunks support an answer.\n\n### How Much Does It Save? Let's Calculate\n\nWe can first get a feel for token savings with a simple estimate.\n\nLet the full document length be $L$, the retrieved-chunk length be $\\ell$, and the fraction judged answerable be $a$. Every question pays the RAG-and-routing cost $\\ell$. Unanswerable questions also pay for the full document $L$. Average input tokens per question are approximately:\n\n$$\n\\ell + (1-a)\\,L\n$$\n\nRelative to pure LC's $L$, that is:\n\n$$\n\\frac{\\ell}{L} + (1-a)\n$$\n\nThe prose gives an example with about 1.5K retrieved tokens and a full document of 10K to 100K tokens. Suppose the document is 100K and the answerable ratio is $a=0.77$:\n\n$$\n\\frac{1.5}{100} + (1-0.77) = 0.015 + 0.23 \\approx 25\\%\n$$\n\nIf $a=0.57$, the ratio becomes about 44%.\n\nThis ignores instruction and output lengths. Its purpose is to show that **savings depend mainly on how often the model is willing to answer from the chunks.**\n\n## Discussion\n\n### Results\n\nReturn to Table 1 and look at the last three rows for each model:\n\n| Model | LC | Self-Route | Answerable ratio | Token ratio |\n| --- | ---: | ---: | ---: | ---: |\n| Gemini-1.5-Pro | 49.70 | 46.41 | 76.78% | 38.39% |\n| GPT-4O | 48.67 | 48.89 | 57.36% | 61.40% |\n| GPT-3.5-Turbo | 32.07 | 35.32 | 74.10% | 38.85% |\n\n*Compiled from Table 1 of the paper. Token ratio is Self-Route's token usage relative to LC.*\n\nGPT-4O uses about 60% of the tokens for almost the same score as LC. Gemini-1.5-Pro uses less than 40%, losing 3.29 points. GPT-3.5-Turbo actually gains 3.25 points over LC, with particularly large gains on the two datasets that exceed 16K: 22.03 versus 14.73 on En.QA and 44.54 versus 34.50 on En.MC.\n\nAll three models judge more than half the questions answerable, supporting the starting idea: most questions do not need the whole document.\n\nGPT-4O's answerable ratio is noticeably lower, so its token ratio is higher. The authors speculate that alignment differences are involved: OpenAI models are more likely to decline answers based only on chunks, yielding fewer answers but higher accuracy among those answered, while Gemini answers directly more often. This is their interpretation; the paper does not test it further.\n\nSwitching to Dragon produces the same trend for Gemini-1.5-Pro: LC 49.70, RAG 38.09, hybrid 46.81, with 77.88% handled by RAG and a token ratio of 37.87% (Table 2).\n\n### How Many Chunks Should We Retrieve?\n\nAs $k$ increases, RAG sees more chunks and approaches LC's performance, but the routing cost per question rises.\n\nThe authors plot the trade-off from $k=1$ to over 50. At $k=1$, RAG scores only 20.24, while Self-Route reaches 37.9: the hybrid's advantage is greatest here. Above 50, all three approaches perform similarly.\n\nThe cost curve is not monotonic. On average, $k=5$ is cheapest: too few chunks send many questions to LC; too many raise every question's routing cost. The cheapest $k$ varies by dataset, however. For datasets such as NarrativeQA and QMSum that do not require multi-hop reasoning, $k=1$ is cheaper.\n\n### Why Does RAG Fail?\n\nThe authors organize questions that RAG marks unanswerable into four causes, then give Gemini-1.5-Pro a few manually annotated examples to classify all failures:\n\n- **A. Multi-step**: The first answer is needed to know what to retrieve next. For example, “What country is the singer of song XXX from?”\n- **B. Too general**: For example, “What does this group think about XXX?” It is difficult to construct a useful retrieval query.\n- **C. Too long and complex**: The retriever struggles to understand the whole question.\n- **D. Implicit**: The answer requires understanding the broader context. In a long conversation about a space voyage, for example, the question asks where the shadow behind the spaceship came from, but the passage explaining it never mentions “shadow.”\n\n[![RAG failure causes across seven LongBench datasets: multi-step, general, complex, implicit, and other](/content/papers/assets/c571b153dec93eecff057696a2888bf48a13750fe10ac943cc49b0c4e2bc9d92.png)](/content/papers/assets/c571b153dec93eecff057696a2888bf48a13750fe10ac943cc49b0c4e2bc9d92.png)\n\n*Figure 4 from the paper, excerpted from [**page 6 of the ACL Anthology PDF**](https://aclanthology.org/2024.emnlp-industry.66.pdf#page=6), CC BY 4.0. Each bar's height is the number of failed RAG questions in that dataset.*\n\nThe causes fit the datasets well.\n\nBlue “multi-step” failures dominate HotpotQA, 2WikiMQA, and MuSiQue, the three Wikipedia multi-hop datasets. NarrativeQA contains long stories rich in dialogue, and green “implicit” almost fills its bar. QMSum asks open-ended questions about meeting transcripts, so red “too general” dominates.\n\nManual inspection of “other” finds that most are also ambiguous multi-step questions.\n\nFor RAG practitioners, this figure suggests a useful starting point: identify which kind of question you have, then decide whether to improve retrieval, rewrite queries, or use long context directly. The authors also suggest trying chain-of-thought within RAG for multi-step questions and revisiting query expansion for general and complex ones.\n\n### The Trap in Synthetic Data\n\nOne final small experiment is especially revealing.\n\n∞Bench's PassKey is a needle-in-a-haystack test: hide “the passkey is 123456” in irrelevant text, then ask “What is the passkey”.\n\n| Question | RAG | LC |\n| --- | ---: | ---: |\n| What is the passkey | 80.34 | 65.25 |\n| What is the special token hidden inside the texts | 4.58 | 69.32 |\n| Two passkeys; ask which is larger | 47.63 | 64.24 |\n\n*Compiled from Table 3 of the paper, using Gemini-1.5-Pro.*\n\nRAG wins decisively with the original wording. Merely rephrasing the question as “What is the special token hidden inside the texts” drops RAG from 80.34 to 4.58, while LC barely changes.\n\nThe likely reason is easy to see: “passkey” appears in both the question and the hidden sentence, giving the retriever an exact match. Change the wording, and that match disappears.\n\nThe authors conclude that the construction of synthetic datasets can unintentionally swing RAG-versus-LC comparisons dramatically. That is why their main evaluation uses real datasets.\n\n### Later Criticism\n\nSelf-Route tries RAG first and falls back to LC on failure. [**Route Before Retrieve**](https://arxiv.org/abs/2605.10235), from May 2026, describes such methods as failure-driven fallbacks: reactive, inefficient, and hard to interpret. It proposes choosing a route before answering, using lightweight document information.\n\nThis criticism highlights an assumption in Self-Route: after reading the chunks, the model must correctly judge whether it can answer. Too optimistic, and RAG returns an incomplete answer; too cautious, and it unnecessarily adds the cost of LC.\n\n## Conclusion\n\nThe paper's answer comes in two halves.\n\n**On accuracy, LC wins.** Across nine real datasets, the average LC score exceeds RAG for Gemini-1.5-Pro, GPT-4O, and GPT-3.5-Turbo, with larger gaps for newer models. The exception is when documents greatly exceed the context window, where RAG remains useful.\n\n**On cost, mixing the two pays off.** Their answers are identical on 63% of questions. Letting the model try retrieved chunks first and use the full document only when it cannot answer gives GPT-4O LC-like performance with about 61% of the tokens. Gemini-1.5-Pro uses about 38%, losing roughly three points.\n\nTwo other findings are worth remembering: RAG failures mainly involve multi-step, general, and implicit questions; and rewording a synthetic test can reverse the winner.\n\nCheck the crib sheet first, then open the whole book if needed. It is a plain idea, but its cost depends on how honestly the model says, “I don't know.”\n","imageSizes":{"/content/papers/assets/bdedfb9a9df07b3ecad6c4f8fee397d8b051787576334f930ad25ea74f7be78e.png":{"width":1289,"height":575},"/content/papers/assets/977e711ec46055e2207ae7f021de5d8f1140e73a8d3dadcc2c3daf7a8d146a43.png":{"width":639,"height":420},"/content/papers/assets/c571b153dec93eecff057696a2888bf48a13750fe10ac943cc49b0c4e2bc9d92.png":{"width":650,"height":381}}}