{"notes":[{"id":"retrieval-augmented-generation/bm25-wins-at-scale","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"BM25 Wins at Scale","subtitle":"Make BM25 Great Again","author":"Z. Yuan","paperDate":"2026-07-29","publishedOn":"2026-09-10","paperTitle":"BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms","paperUrl":"https://arxiv.org/abs/2607.26497","bibliography":{"title":"BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms","authors":["Pengyu Wang","Benfeng Xu","Shaohan Wang","Mingxuan Du","Xin Zeng","Huarui Wu","Lei Zhang","Licheng Zhang"],"url":"https://arxiv.org/abs/2607.26497","pdfUrl":"https://arxiv.org/pdf/2607.26497v3","codeUrl":null,"preprint":{"id":"2607.26497","submittedOn":"2026-07-29","url":"https://arxiv.org/abs/2607.26497"},"publication":null,"citation":"@misc{retrievalaugmentedgenerationbm25winsatscale2026,\n  title = {{BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms}},\n  author = {Pengyu Wang and Benfeng Xu and Shaohan Wang and Mingxuan Du and Xin Zeng and Huarui Wu and Lei Zhang and Licheng Zhang},\n  year = {2026},\n  eprint = {2607.26497},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2607.26497}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/bm25-wins-at-scale.faa6780c7835.bib"},"thumbnail":{"src":"/content/papers/assets/d96c5fc056dc2c5c1af9de8338acd25396a1e0a15973938281502d305501c18e.png","width":700,"height":550},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/bm25-wins-at-scale.536f11d8ba29.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"a-fictional-companys-knowledge-base","text":"A Fictional Company's Knowledge Base","depth":3},{"id":"the-foundation-1144-fixed-documents","text":"The Foundation: 1,144 Fixed Documents","depth":3},{"id":"twenty-eight-nested-databases","text":"Twenty-Eight Nested Databases","depth":3},{"id":"seven-pipelines-four-schools","text":"Seven Pipelines, Four Schools","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"main-results-a-crossover-near-ten-million-tokens","text":"Main Results: A Crossover Near Ten Million Tokens","depth":3},{"id":"cost-the-agents-search-gets-more-expensive","text":"Cost: The Agent's Search Gets More Expensive","depth":3},{"id":"graph-based-rag-hits-the-index-building-wall","text":"Graph-Based RAG Hits the Index-Building Wall","depth":3},{"id":"why-does-the-agent-lose-find-it-before-you-reason-about-it","text":"Why Does the Agent Lose? Find It Before You Reason About It","depth":3},{"id":"where-does-bm25s-advantage-come-from","text":"Where Does BM25's Advantage Come From?","depth":3},{"id":"what-to-keep-in-mind","text":"What to Keep in Mind","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"By 2026, RAG had split into several schools.\n\n**Lexical retrieval** ranks keywords with methods such as BM25. **Dense retrieval** converts text into vectors and compares similarity. **Graph-based RAG** asks an LLM to extract entities and relationships during indexing and builds a knowledge graph. Then there are **agents** that work like coding agents, using `ls`, `grep`, and file reading to search a directory tree step by step.\n\nEvery school has impressive paper results. The problem is that these results usually come from **different benchmarks**, tested at **one particular data size**.\n\nEnterprise knowledge bases do not stay one size. Tens of thousands of documents today may become hundreds of thousands next year.\n\nThis preprint from the University of Science and Technology of China, Metastone, and the Information Technology Research Center of the Beijing Academy of Agriculture and Forestry Sciences asks a practical question: **as the data keeps growing, which RAG approach holds up?**\n\nThe paper was first uploaded to arXiv on July 29, 2026 under the title “Which RAG Paradigm Wins at Scale?” before adopting its current title. This note follows v3, dated July 31, 2026.\n\n\n> [!TIP]\n> \n> **What Is BM25?**\n> \n> BM25 is a keyword-ranking method that matured in the 1990s. Search engines and tools such as Elasticsearch still use it as a default ranking option.\n> \n> It scores documents using roughly three considerations:\n> \n> 1. **How often query terms appear.** More occurrences raise the score, but the gain saturates: a hundred occurrences do not score ten times as much as ten.\n> 2. **How rare a term is.** “The” appears everywhere and contributes little. “ticket-4821” appears in few documents, so a match matters.\n> 3. **How long the document is.** Long documents naturally match more often, so length is adjusted for.\n> \n> It needs neither training nor a GPU. Build an inverted index and it is ready to use.\n\n\n## Defining the Problem\n\nThe authors define scaling precisely: **hold the questions fixed, grow only the database, and measure how each method changes.**\n\nThat requires three things:\n\n1. **Fixed questions and answers.** Changing questions would mix data growth with task difficulty.\n2. **Fixed distractors.** Deliberately misleading documents should be present from the start, rather than introduced gradually as the corpus grows.\n3. **Fixed comparison conditions.** The answering model, scoring procedure, and cost accounting must be consistent across methods.\n\n## Solving the Problem\n\n### A Fictional Company's Knowledge Base\n\nThe authors use EnterpriseRAG-Bench, which simulates a fictional LLM inference-service company. It contains 511,957 documents totaling 600.8 million tokens across nine sources, including wikis, chat threads, tickets, email, meeting transcripts, CRM records, and code reviews.\n\nThere are 500 questions in ten categories, ranging from basic lookup to exhaustive information gathering, resolving contradictory information, and answering high-level questions. Each question has a reference answer, supporting documents—722 in total—and atomic facts the answer should contain.\n\nThe corpus already includes 7.7% noisy documents: misplaced sources or paths, or outdated near-duplicates, much like a real company's files.\n\n### The Foundation: 1,144 Fixed Documents\n\nThe authors first build a minimal database, called the **bedrock**:\n\n$$\n\\underbrace{722}_{\\text{gold documents}}+\\underbrace{326}_{\\text{traps}}+\\underbrace{99}_{\\text{lures}}+\\underbrace{2}_{\\text{company overviews}}-\\underbrace{5}_{\\text{duplicates}}=1{,}144\n$$\n\n**Traps** are crucial: they cover the same topic as a gold document but contain the wrong version, date, or decision—for example, an old specification for the same project. The authors take the top ten BM25 results from the full corpus, then use dense retrieval to rerank BM25's top 200 and select another ten. An LLM filters these for “same topic, wrong facts.”\n\n**Lures** are prepared for unanswerable questions: documents similar to the question but verified not to answer it. This prevents a model from getting the question right merely because it found nothing.\n\nAll these distractors are included in the smallest database and remain unchanged thereafter.\n\n### Twenty-Eight Nested Databases\n\nThe remaining documents are placed in a fixed order. Starting from the bedrock, each level adds documents so that the count grows by a factor of 1.25:\n\n$$\nn_{t+1}\\approx1.25\\,n_t\n$$\n\nThe corpus grows from 1,144 to 511,959 documents across 28 levels, roughly 450-fold, or from 1.7 million to 601 million tokens.\n\n“Nesting” is strict: every level fully contains the preceding one, verified with file checksums. From level 1 to level 28, the only change is the addition of irrelevant background documents.\n\nAdded documents are stratified by source and noise proportion, keeping each level's composition close to the full corpus.\n\n### Seven Pipelines, Four Schools\n\n| School | Method | How it works |\n| --- | --- | --- |\n| Lexical | BM25 | Inverted index; no LLM required |\n| Dense | DenseRAG | Embed each chunk and compare similarity |\n| Graph | HippoRAG 2 | Extract triples, build a graph, retrieve with Personalized PageRank |\n| Graph | MS-GraphRAG | Entity graph and hierarchical community reports; local search |\n| Graph | LightRAG | Two-level entity–relation index; hybrid mode |\n| Graph | LinearRAG | Lightweight named-entity recognition builds a co-occurrence graph without generative LLM calls |\n| Agent | File-System Agent | No index; list directories, search, and read the raw file tree |\n\n*Compiled from §4.1 of the paper.*\n\nFile-System Agent uses a coding-agent-style interface, with up to 80 LLM calls per question.\n\nOther conditions are aligned as closely as possible:\n\n- **Answering model**: Qwen3.6-27B at temperature 0 for every method. File-System Agent uses the same model to choose its next action.\n- **Embeddings**: Qwen3-Embedding-0.6B throughout.\n- **Retrieval depth**: Methods with a retrieval-depth setting take the top five chunks. BM25, DenseRAG, and HippoRAG 2 share the same chunking scheme.\n- **Cost accounting**: Every LLM call is intercepted, and its tokens assigned to either indexing or answering.\n- **Scoring**: The benchmark's official protocol uses **combined score**, measuring factual completeness after a correctness gate. Each question is run once; confidence intervals are estimated by bootstrapping questions.\n\nAn independent judge model rescoring the answers agrees with the original in 96.2% of judgments. A simpler binary correct/incorrect score also preserves the ranking at all nine shared levels.\n\n## Discussion\n\n### Main Results: A Crossover Near Ten Million Tokens\n\n[![Official combined score versus corpus tokens, with 95% confidence intervals; BM25 and File-System Agent cross near ten million tokens](/content/papers/assets/d96c5fc056dc2c5c1af9de8338acd25396a1e0a15973938281502d305501c18e.png)](/content/papers/assets/d96c5fc056dc2c5c1af9de8338acd25396a1e0a15973938281502d305501c18e.png)\n\n*Figure 3 from the paper, excerpted from [**page 6 of the v3 PDF**](https://arxiv.org/pdf/2607.26497v3#page=6). The horizontal axis is corpus tokens on a logarithmic scale; the vertical axis is official combined score. Bands show 95% confidence intervals. Graph-method curves stop at the largest level that could be built.*\n\nThe horizontal axis shows corpus tokens logarithmically; the vertical axis shows score.\n\nLook at the top two curves. At the left, cyan File-System Agent is slightly above brown BM25: 77.4 versus 74.7 at the bedrock. But their 95% confidence intervals—73.9–80.8 and 71.4–77.9—overlap, so they are not statistically distinguishable.\n\nMoving right, the curves cross around $10^7$, or ten million tokens. From there, BM25 leads at every shared level, with an increasingly large gap.\n\nAt the full corpus on the far right:\n\n| Method | 1,144 documents (bedrock) | 21,614 documents | 131,876 documents | 511,959 documents (full) |\n| --- | ---: | ---: | ---: | ---: |\n| BM25 | 74.7 | 64.9 | 55.2 | **50.5** |\n| File-System Agent | **77.4** | 62.6 | 50.9 | 30.7 |\n| DenseRAG | 58.1 | 44.2 | 36.0 | 29.9 |\n| HippoRAG 2 | 66.2 | 53.8 | 41.0 | — |\n| LinearRAG | 46.2 | 34.3 | 29.8 | — |\n| MS-GraphRAG | 45.9 | — | — | — |\n| LightRAG | 48.0 | — | — | — |\n\n*Compiled from Table 1 of the paper. Columns are document counts at different levels. “—” means the level was not built or evaluated.*\n\nAt full scale, BM25 scores 50.5, File-System Agent drops to 30.7, and DenseRAG reaches 29.9. The gap is nearly 20 points.\n\nDense retrieval stays below BM25 throughout. Most graph methods never reach the full corpus at all.\n\n### Cost: The Agent's Search Gets More Expensive\n\nFor one-retrieval-then-answer pipelines such as BM25, DenseRAG, and HippoRAG 2, query cost barely changes with corpus size: roughly 5,000 to 6,000 tokens per question, mostly the answering model's prompt.\n\nFile-System Agent is different. It already needs 226K tokens per question at the bedrock, 39 times BM25's cost. At 21,614 documents, that rises to 343K, about 60 times as much.\n\nMedian LLM calls rise from five to eight at 42,587 documents. The fraction exhausting the 80-call budget stays below 7% through 42,587 documents, rises to 15% at 131,876, and reaches 31% at full scale.\n\nThe authors stress that accuracy also declines among questions that stay within budget. The collapse cannot be attributed solely to too small a budget.\n\nFor 500 questions, the arithmetic is revealing: BM25 at 5.8K tokens per question spends 2.9 million tokens for 74.7 points. File-System Agent at 226K spends about 113 million tokens for 77.4. Nearly 40 times the tokens buy 2.7 points, still within overlapping confidence intervals.\n\n### Graph-Based RAG Hits the Index-Building Wall\n\nFor graph methods, most cost comes from indexing: an LLM reads each passage and extracts entities and relations.\n\nThe authors fit $C(x)=a\\,x^{b}$ to indexing tokens versus corpus size, then extrapolate to the full corpus:\n\n| Method | Exponent $b$ | Full-corpus extrapolation | Estimated single-machine time |\n| --- | ---: | ---: | ---: |\n| HippoRAG 2 | 1.01 | About 2.9 billion tokens | About 3 days |\n| MS-GraphRAG | 0.92 | About 7.9 billion tokens | About 50 days |\n| LightRAG | 1.36 | About 102 billion tokens | About 4 years |\n\n*Compiled from Table 2 of the paper.*\n\nLightRAG's exponent exceeds one: indexing cost per document rises as the corpus grows. The authors attribute this to repeated entity rewriting and merging. It already fails to finish within their resources at 2,826 documents.\n\nEven completed indexes score poorly. On a corpus of about 155 million tokens, HippoRAG 2 spends 724 million tokens building its index and scores 41.0, about 15 points below BM25 at that level.\n\nThe authors identify three disadvantages for graph methods on this corpus: extraction introduces noise, including many fragmentary entities among the bedrock's 32,000; graph proximity favors semantically related but factually wrong material, exactly what trap documents exploit; and HippoRAG 2 discards extracted predicate text, losing relationship semantics.\n\n### Why Does the Agent Lose? Find It Before You Reason About It\n\nThis is the paper's key controlled experiment.\n\nDoes File-System Agent lose because it cannot find the evidence, or because it cannot use evidence once found?\n\nOn a fixed set of 150 questions, the authors compare the bedrock and full corpus. A new condition, **Agent+BM25**, keeps the model, harness, 80-call budget, and judge unchanged. Only the raw file-tree tools are replaced by BM25-ranked search and chunk reading, with the first search forced to use the original question.\n\n| Method | Bedrock | Full | Document recall | Calls | Tokens |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| Native BM25 | 81.3 | 54.8 | 65.6 | 1.00 | 5.8K |\n| File-System | 87.1 | 36.9 | 36.8 | 36.12 | 895K |\n| Agent+BM25 | **90.1** | **69.4** | **72.4** | 5.79 | 101K |\n\n*Compiled from Table 4: the same 150 questions, rescored together. The first two score columns give combined score at the two levels. Document recall, calls per question, and tokens per question refer to the full corpus.*\n\nThe answer is clear: **the problem is finding the evidence.**\n\nAt full scale, File-System Agent hits at least one gold document on only 39.0% of questions; BM25 does so on 71.6%.\n\nAmong questions where each method finds at least one gold document, File-System Agent scores 85.9 versus BM25's 73.8. Once it finds information, the agent appears better at reasoning and synthesis. But these are different subsets, defined by each method's own hits, so the numbers are descriptive and cannot establish a direct ranking.\n\nChanging only the retrieval tools raises Agent+BM25 to 69.4 on the full corpus, 32.5 points above the original File-System Agent and 14.6 above Native BM25. Its hit rate reaches 78.0%, while tokens per question fall from 895K to 101K—about one ninth.\n\nThe authors conclude that **agent reasoning works best after global ranking, rather than as its replacement**. As the file tree grows, navigating it branch by branch makes the relevant branches harder to reach.\n\n\n> [!NOTE]\n> \n> **The Harness Matters Too**\n> \n> On 150 bedrock questions, the authors compare three harnesses that read raw files: their own scores 86.3, Pi-Agent 82.3, and Codex only 43.9. Native BM25 scores 82.1 (Table 5).\n> \n> With the same model, corpus, and judge, changing the harness moves scores by over 40 points. The Agent+BM25 comparison above uses the best of those three: the authors' implementation.\n\n\n### Where Does BM25's Advantage Come From?\n\nThe authors acknowledge that this corpus favors BM25. Enterprise questions often include exact lexical anchors—project names, ticket IDs, and people—while traps are semantically similar but factually wrong. Exact matching is less easily fooled.\n\nThey run two controls:\n\n- **Paraphrase the questions** to reduce lexical overlap. At the bedrock, BM25 falls to 63.9, File-System Agent scores 73.3, and DenseRAG 51.8. Paraphrasing strengthens the agent's small-corpus advantage, but BM25 still beats dense and graph retrieval.\n- **Retrieve ten chunks** instead of five. BM25 scores 83.0 and DenseRAG 70.0; the gap remains.\n\n### What to Keep in Mind\n\n1. **One benchmark.** EnterpriseRAG-Bench is a simulated enterprise corpus. Its lexical anchors and trap design affect lexical-versus-semantic retrieval performance.\n2. **A small dense model.** All embeddings come from Qwen3-Embedding-0.6B. The paper does not test whether larger embedding models would change DenseRAG's results.\n3. **No hybrid-retrieval evaluation.** It compares individual pipelines, without common combinations such as BM25 fused with dense retrieval or followed by a reranker. Agent+BM25 is the closest.\n4. **One answering model and one run per question.** Uncertainty comes from resampling questions, without run-to-run variation.\n5. **Code and data.** The appendix describes supplementary code and data and how they correspond, but the v3 main text provides no public download location.\n\n## Conclusion\n\nAcross 28 nested databases spanning roughly 450-fold growth, the authors hold questions, gold documents, and traps fixed and add only background documents. Under this setup:\n\n1. **There is a crossover, not a permanent winner.** File-System Agent has a slight edge at small scale. Beyond about ten million tokens, BM25 leads at every shared level, by nearly 20 points at full scale.\n2. **BM25 is cheap and steady.** At about 5.8K tokens per question and with no LLM indexing cost, it occupies the low-cost end of the cost–accuracy trade-off.\n3. **The agent's problem is finding evidence.** Replacing file-tree tools with BM25-ranked search raises the same agent from 36.9 to 69.4 at full scale, using about one ninth of the tokens.\n4. **Graph-based RAG gets stuck at indexing.** LightRAG extrapolates to about four years on one machine. Even HippoRAG 2, which builds at larger sizes, scores below BM25.\n\nThe authors' practical recommendation for enterprise corpora is direct: **use BM25 by default; use Agent+BM25 when questions require synthesizing multiple sources**. LLM-built graph indexes are difficult to justify at hundreds of thousands to millions of documents unless construction approaches linear cost and questions predominantly concern relationships.\n\nWhen the data grows, a thirty-year-old method returns. This time, it stands in front of the agent, clearing a path.\n","imageSizes":{"/content/papers/assets/d96c5fc056dc2c5c1af9de8338acd25396a1e0a15973938281502d305501c18e.png":{"width":700,"height":550}}}