{"notes":[{"id":"retrieval-augmented-generation/is-grep-all-you-need","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Is Grep All You Need","subtitle":"Is Grep Enough?","author":"Z. Yuan","paperDate":"2026-05-14","publishedOn":"2026-08-29","paperTitle":"Is Grep All You Need? How Agent Harnesses Reshape Agentic Search","paperUrl":"https://arxiv.org/abs/2605.15184","bibliography":{"title":"Is Grep All You Need? How Agent Harnesses Reshape Agentic Search","authors":["Sahil Sen","Akhil Kasturi","Elias Lumer","Anmol Gulati","Vamse Kumar Subbiah"],"url":"https://arxiv.org/abs/2605.15184","pdfUrl":"https://arxiv.org/pdf/2605.15184v1","codeUrl":null,"preprint":{"id":"2605.15184","submittedOn":"2026-05-14","url":"https://arxiv.org/abs/2605.15184"},"publication":null,"citation":"@misc{retrievalaugmentedgenerationisgrepallyouneed2026,\n  title = {{Is Grep All You Need? How Agent Harnesses Reshape Agentic Search}},\n  author = {Sahil Sen and Akhil Kasturi and Elias Lumer and Anmol Gulati and Vamse Kumar Subbiah},\n  year = {2026},\n  eprint = {2605.15184},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2605.15184}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/is-grep-all-you-need.4ebb3c804df8.bib"},"thumbnail":{"src":"/content/papers/assets/7388f9c6a8dfcf342dfed65f8f8c3d27d2ffbbf3d5dfb51aeb45005be29a2c00.png","width":944,"height":422},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/is-grep-all-you-need.1e37b0d423ef.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-task-recover-memories-from-long-conversations","text":"The Task: Recover Memories from Long Conversations","depth":3},{"id":"two-retrieval-tools","text":"Two Retrieval Tools","depth":3},{"id":"four-harnesses","text":"Four Harnesses","depth":3},{"id":"two-ways-to-deliver-results","text":"Two Ways to Deliver Results","depth":3},{"id":"models-and-scoring","text":"Models and Scoring","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"experiment-1-grep-harness-and-delivery","text":"Experiment 1: Grep, Harness, and Delivery","depth":3},{"id":"experiment-2-what-happens-with-more-distractors","text":"Experiment 2: What Happens with More Distractors?","depth":3},{"id":"the-two-experiments-numbers-do-not-match","text":"The Two Experiments' Numbers Do Not Match","depth":3},{"id":"the-authors-own-boundaries","text":"The Authors' Own Boundaries","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"If you have used Claude Code, Codex, or Gemini CLI over the past two years, you have probably seen this: you ask a question about code, the agent runs a series of `grep` searches, reads a few files, then comes back with an answer.\n\nThat differs from the standard RAG setup of recent years. The default was to split documents into chunks, convert them into vectors, store them in a vector database, and search by semantic similarity.\n\nSo people started asking: if an agent can grep for itself, do we still need vector retrieval?\n\nThis preprint from PwC in the United States appeared on arXiv in May 2026. Its title asks whether grep is enough. Read it through, though, and the authors have measured something else: **the same retrieval tool can perform very differently inside different agent harnesses.**\n\n\n> [!TIP]\n> \n> **What Is a Harness?**\n> \n> The paper repeatedly uses **agent harness**. Think of it as the program wrapped around the model: it chooses available tools, writes the system prompt, formats tool results, and decides when to stop.\n> \n> Claude Opus 4.6 running inside Claude Code and the same model inside a homemade LangChain agent use different harnesses. The model stays the same, while prompts, tool descriptions, and return formats differ.\n\n\n## Defining the Problem\n\nThe authors argue that earlier retrieval comparisons mostly assume a fixed pipeline: retrieve documents, put them into the prompt, and have the model answer.\n\nAgents use retrieval differently. They decide what to search for, how many times to search, and whether the results are sufficient. If not, they try another keyword. The harness affects all of this.\n\nThe authors therefore ask three questions:\n\n1. **Which works better across harnesses: grep or vector retrieval?**\n2. **Does the answer change when tool results are inserted into the conversation versus written to files for the agent to read?**\n3. **How does performance change as irrelevant information grows?**\n\n## Solving the Problem\n\n### The Task: Recover Memories from Long Conversations\n\nThe evaluation uses a 116-question subset of LongMemEval.\n\nLongMemEval tests long-term conversational memory. Each question comes with many chat sessions; one or more contain the answer, while the rest are irrelevant distractors. Questions fall into six categories:\n\n| Category | What it tests |\n| --- | --- |\n| knowledge-update | Tracking state changes over time |\n| multi-session | Combining information across conversations |\n| single-session-assistant | Recalling something the assistant said |\n| single-session-preference | The user's personal preferences |\n| single-session-user | Facts stated by the user |\n| temporal-reasoning | Calculating durations, ordering events, and inferring dates |\n\nFor an illustrative example of our own, a question might ask, “How many months ago did I last change jobs?” The answer is hidden in one sentence among dozens of chat sessions.\n\nThe authors additionally apply their earlier Chronos preprocessing to extract dates and time intervals into structured event records. These are stored alongside the original conversations in separate files for each question. Their rationale is that they want to test finding evidence, rather than reconstructing dates from fragmented conversations.\n\n### Two Retrieval Tools\n\n**grep** loads each question's conversations and event records into memory, matches the raw text with regular expressions, and ranks results by hit count. It needs no embedding model, vector index, or external service.\n\n**Vector retrieval** embeds each conversation turn and event ahead of time and stores them in a question-specific index. At query time, it embeds the question, retrieves candidates through approximate nearest-neighbor search, reranks them, and returns the top $k$ items. The agent chooses $k$.\n\nThe paper does not name the embedding model or reranker.\n\n### Four Harnesses\n\n- **Chronos**: The authors' LangChain agent has four search tools—grep and vector versions for conversations and events—enabled or disabled by experimental condition. Its system prompt changes dynamically by question category, with different instructions for temporal reasoning and preferences, for example.\n- **Claude Code**, **Codex CLI**, and **Gemini CLI**: The model providers' own command-line agents receive the question and a dynamically generated search strategy, then call wrapped grep or vector-search scripts through bash.\n\n\n> [!NOTE]\n> \n> **One Setup Detail to Keep in Mind**\n> \n> According to §3.3.1, Chronos begins each question with initial context containing the top 15 vector-retrieved memories, before entering the tool-calling loop. The paper does not clarify whether this step is removed in the grep-only setting. Keep that ambiguity in mind when reading Chronos's grep scores.\n\n\n### Two Ways to Deliver Results\n\n- **Inline (standard)**: Print search results to standard output or insert them into the conversation as a tool response. The model sees them immediately, but large results occupy context.\n- **Programmatic (file-based)**: Write results to a file and return only its location. The model must make another tool call to read or grep that file. This avoids flooding the context, at the cost of an extra step.\n\n### Models and Scoring\n\nFive models are tested: Claude Opus 4.6, Claude Haiku 4.5, GPT-5.4, Gemini 3.1 Pro, and Gemini 3.1 Flash-Lite.\n\nFollowing LongMemEval's evaluation procedure, GPT-4o judges correctness from the question, reference answer, and agent response.\n\n## Discussion\n\n### Experiment 1: Grep, Harness, and Delivery\n\n[![Table 1: accuracy of grep and vector retrieval with inline and programmatic delivery, for five models in Chronos and their respective CLIs](/content/papers/assets/7388f9c6a8dfcf342dfed65f8f8c3d27d2ffbbf3d5dfb51aeb45005be29a2c00.png)](/content/papers/assets/7388f9c6a8dfcf342dfed65f8f8c3d27d2ffbbf3d5dfb51aeb45005be29a2c00.png)\n\n*Table 1 from the paper, excerpted from [**page 5 of the v1 PDF**](https://arxiv.org/pdf/2605.15184v1#page=5), CC BY 4.0. There are 116 questions with the full distractor set. The first two result columns are inline; the last two are programmatic.*\n\nEach row fixes a model–harness pair. Four columns cross two retrieval methods with two delivery methods.\n\n**First, the inline columns.** Grep beats vector retrieval in all ten combinations. The largest gap is Chronos with Gemini 3.1 Flash-Lite: 86.2% versus 62.9%. The smallest is Claude Code with Claude Opus 4.6: 76.7% versus 75.0%.\n\n**Next, the same model in different harnesses.** Claude Opus 4.6 with inline grep scores 93.1% in Chronos but only 76.7% in Claude Code, a 16.4-percentage-point gap. Within Chronos, the grep-versus-vector gap is 9.5 points. Changing the harness can matter as much as changing the retriever, or more.\n\n**Finally, the programmatic columns.** With file-based delivery, vector retrieval overtakes grep in five of ten combinations. The most dramatic case is Codex CLI with GPT-5.4: inline grep scores 93.1%, tied for best. Under programmatic delivery, grep falls to 55.2%, and vector retrieval reaches only 67.2%.\n\nThe authors offer two interpretations:\n\n1. **LongMemEval rewards literal evidence.** Exact dates, counts, preferences, and original statements remain stable strings through tokenization. Grep finds them directly, without passing through embeddings.\n2. **File-based delivery makes the task harder.** “Read the tool response” becomes “locate a file, open it, and integrate its contents.” If the extra step is unreliable, accuracy can collapse independently of retrieval quality.\n\n\n> [!TIP]\n> \n> **With 116 Questions, How Much Is One Answer Worth?**\n> \n> One question is $1/116\\approx0.86$ percentage points.\n> \n> Claude Code with Opus scores 76.7% versus 75.0%, or 89 correct answers versus 87—a difference of just two. Each condition appears to have been run once, and the paper reports no variability range. Comparisons separated by only a few answers should carry limited weight.\n\n\n### Experiment 2: What Happens with More Distractors?\n\nThe second experiment varies the number of sessions visible per question: 5, 10, 20, 30, and the full 39–66. Answer-containing sessions are always retained; the remaining slots are filled with distractor sessions for that question.\n\nA common intuition is that grep suffices for small data, while larger data needs semantic search.\n\nThe results are less simple:\n\n- **The curves are not monotonic.** For Chronos with Opus, grep drops from 90.5% at s20 to 85.3% at s30, then rises to 89.7% with the full data.\n- **Vectors often do better with little data.** At s5, Chronos's grep scores range from 83.2% to 89.7%, versus 87.9% to 94.0% for vectors.\n- **The winner depends on the harness.** Claude Code with Opus or Haiku favors grep in every setting. Gemini CLI with Gemini 3.1 Pro favors vectors in every setting, reaching 89.7% versus 78.5% with full data. Chronos alternates winners across conditions.\n\nThe authors also note that distractor sessions are resampled for each session limit. A mid-curve fluctuation such as “30 is easier than 20” may come from that particular distractor sample. Interpret these curves with the effects of random sampling in mind.\n\n### The Two Experiments' Numbers Do Not Match\n\nOne issue needs particular attention.\n\nThe full-data settings in Experiments 1 and 2 appear to use the same 116 questions, distractors, and inline delivery, yet their numbers differ substantially:\n\n| Model and harness | Retrieval | Experiment 1 (Table 1) | Experiment 2 full (Tables 2, 3) |\n| --- | --- | ---: | ---: |\n| Opus + Claude Code | grep | 76.7 | 94.0 |\n| Haiku + Claude Code | grep | 55.2 | 81.9 |\n| Opus + Chronos | Vector | 83.6 | 92.2 |\n| Gemini 3.1 Pro + Gemini CLI | grep / vector | 81.9 / 75.0 | 78.5 / 89.7 |\n\nThe last row even reverses the winner: grep leads in Experiment 1, vectors in Experiment 2. The paper does not explain the setup difference. Experiment 2 also lacks Codex rows, which the authors mark as pending.\n\nStrictly speaking, the conclusion that grep consistently outperforms vectors holds only for Experiment 1's inline setting.\n\n### The Authors' Own Boundaries\n\nThe limitations section is clear: these conclusions concern long-term conversational-memory QA. Evidence in these tasks is often a sentence that can be found verbatim, favoring literal matching. Results may differ for paraphrased summaries of scientific literature, image-heavy documents, or code semantics, where evidence rarely appears in exactly the same wording.\n\nThe authors explicitly **avoid claiming that grep universally beats vector retrieval**. They claim that grep can win end-to-end comparisons on the tasks and corpus they studied.\n\n## Conclusion\n\nThe title asks whether grep is enough. The answer is: **often, for this task and these settings—but it depends on who uses it and how.**\n\n1. **Inline grep wins all ten combinations.** Long-conversation evidence is often a string that can be found verbatim, exactly grep's strength.\n2. **The harness matters as much as the retrieval method.** The same Claude Opus 4.6 differs by 16.4 percentage points between the homemade harness and Claude Code.\n3. **File-based delivery reshuffles the winners.** Grep with Codex and GPT-5.4 falls from 93.1% to 55.2%; vectors lead in five of ten combinations.\n4. **There is no fixed winner as distractors grow.** Results vary with harness, model, and sampling.\n\nThese findings come from 116 questions, single runs, an unnamed embedding model, and experiments with conflicting numbers. They suggest directions to validate yourself, rather than a settled verdict.\n\nThe authors end with useful advice: evaluate the retrieval mechanism, harness orchestration, and result-delivery path as one system.\n\nTo compare grep with vectors, run the comparison inside your own agent.\n","imageSizes":{"/content/papers/assets/7388f9c6a8dfcf342dfed65f8f8c3d27d2ffbbf3d5dfb51aeb45005be29a2c00.png":{"width":944,"height":422}}}