{"notes":[{"id":"retrieval-augmented-generation/deep-agentic-search","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Deep Agentic Search","subtitle":"A Game of Telephone","author":"Z. Yuan","paperDate":"2026-08-02","publishedOn":"2026-09-27","paperTitle":"Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study","paperUrl":"https://arxiv.org/abs/2608.01507","bibliography":{"title":"Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study","authors":["Amirkia Rafiei Oskooei","Bora Ilci","Alperen Kayim","Mehmet Egemen Uzun","Berat Can","Kaan Emre Kara","Ozan Orhan","Mehmet S. Aktas"],"url":"https://arxiv.org/abs/2608.01507","pdfUrl":"https://arxiv.org/pdf/2608.01507v1","codeUrl":null,"preprint":{"id":"2608.01507","submittedOn":"2026-08-02","url":"https://arxiv.org/abs/2608.01507"},"publication":null,"citation":"@misc{retrievalaugmentedgenerationdeepagenticsearch2026,\n  title = {{Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study}},\n  author = {Amirkia Rafiei Oskooei and Bora Ilci and Alperen Kayim and Mehmet Egemen Uzun and Berat Can and Kaan Emre Kara and Ozan Orhan and Mehmet S. Aktas},\n  year = {2026},\n  eprint = {2608.01507},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2608.01507}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/deep-agentic-search.1f8bfd9bac55.bib"},"thumbnail":{"src":"/content/papers/assets/61ecd2fcf8409a060dc66b99123841b7e1bceff328c5f1ee011397bb73773c74.png","width":944,"height":427},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/deep-agentic-search.fd5189728bb1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"two-architectures","text":"Two Architectures","depth":3},{"id":"implementation-details","text":"Implementation Details","depth":3},{"id":"models-data-and-scoring","text":"Models, Data, and Scoring","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"accuracy-semantic-wins-for-every-model","text":"Accuracy: Semantic Wins for Every Model","depth":3},{"id":"cost-more-spending-does-not-buy-higher-accuracy","text":"Cost: More Spending Does Not Buy Higher Accuracy","depth":3},{"id":"most-of-deeps-work-happens-outside-the-main-agents-view","text":"Most of Deep's Work Happens Outside the Main Agent's View","depth":3},{"id":"failure-analysis-handoffs-are-the-weak-spot","text":"Failure Analysis: Handoffs Are the Weak Spot","depth":3},{"id":"the-authors-own-boundaries","text":"The Authors' Own Boundaries","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"Before answering “Where is this feature implemented?”, today's coding agents often send out a helper.\n\nInstead of browsing files itself, the main agent asks a **subagent** to explore: list directories, grep, read files, and keep searching. The subagent works in its own context and returns only an organized conclusion. The main agent's context stays clean, containing just the concise result.\n\nThis design is called **Deep Agentic Search**, sometimes described as “grep search performed by a subagent.” The paper notes its rapid adoption by coding agents such as Claude Code, Codex, and Antigravity.\n\nBut is it actually more accurate?\n\nThis empirical study from Yildiz Technical University and Intellica Business Intelligence in Istanbul, Türkiye, appeared on arXiv in August 2026. The authors compare it with a more traditional approach: build a vector index ahead of time and let the agent retrieve directly, using the same code question-answering tasks.\n\nThe result is surprising: the traditional approach passes 65.2% of questions, versus 46.2% for Deep Agentic Search, whose cost per correct answer is more than twice as high.\n\n\n> [!TIP]\n> \n> **Why Delegate Search to a Subagent?**\n> \n> The design addresses **context pollution**, often called context rot: as an agent searches and reads files, irrelevant code and search results fill its context and degrade its judgment.\n> \n> Delegating messy exploration while keeping only concise conclusions in the main context is considered good **context engineering** practice. This paper tests its actual effect on answer accuracy.\n\n\n## Defining the Problem\n\nThe authors ask three questions:\n\n1. **Accuracy**: Which approach answers repository-wide questions better?\n2. **Cost**: How much does each cost, and does additional spending buy accuracy?\n3. **Behavior and failure**: How do the approaches operate, and how do they fail?\n\n## Solving the Problem\n\n### Two Architectures\n\n[![Two search paradigms: a ReAct agent retrieves code chunks from a prebuilt vector index above; a Deep Agent delegates exploration to a subagent in a separate context, without an index, below](/content/papers/assets/44afeaa4b2a8e82e998b7d93f70c43ca555a0c6588ac386081af2aedc5612536.png)](/content/papers/assets/44afeaa4b2a8e82e998b7d93f70c43ca555a0c6588ac386081af2aedc5612536.png)\n\n*Fig. 1 from the paper, excerpted from [**page 2 of the v1 PDF**](https://arxiv.org/pdf/2608.01507v1#page=2). (a) Semantic search implemented with a ReAct agent; (b) Deep Agentic search implemented with a Deep Agent.*\n\n**Upper half (a): Semantic search.** The repository is split into chunks, embedded, and stored in a vector database beforehand. When a question arrives, the ReAct agent turns it into a query vector, retrieves relevant code chunks, and assembles its own answer.\n\n**Lower half (b): Deep Agentic search.** There is no index. The Deep Agent receives a question and sends an Explorer Subagent into the repository. In the figure, the subagent says “Not here” in one folder, then “Found it!” in another. It returns its findings to the main agent, which answers.\n\nNotice the dashed arrows labeled “Waiting”: while the main agent waits, it cannot see what the subagent actually does.\n\n### Implementation Details\n\nBoth architectures use LangChain.\n\n**Semantic search** uses LangChain's ReAct agent, with only the three tools in SWE-QA's original evaluation protocol: inspect repository structure, perform vector search, and read files. It has no shell, file-writing capability, or planning tool.\n\nThe appendix specifies the vector index:\n\n- A code-aware `RecursiveCharacterTextSplitter` creates 500-token chunks with 50-token overlap.\n- OpenAI's `text-embedding-3-small` produces 1,536-dimensional embeddings.\n- Chroma stores a separate vector database per repository, using cosine similarity and retrieving the top ten chunks each time.\n\n**Deep Agentic search** uses LangChain's Deep Agents framework with its default tools: write a to-do list (`write_todos`), delegate to subagents (`task`), `grep`, `ls`, `glob`, and read files. Main agents and subagents use the same model.\n\n\n> [!NOTE]\n> \n> **Is It Fair to Compare Different Tools?**\n> \n> The authors address this directly: the tool differences define the two paradigms. Add planning and subagents to the ReAct agent and it becomes a Deep Agent; remove those tools from Deep Agent and substitute a vector index, and it becomes the ReAct agent.\n> \n> They are comparing two complete paradigms, rather than an ablation varying only one factor. Everything else is held fixed: the same 15 repositories, 720 questions, four models, judge, and scoring rubric. Each model is compared with itself.\n\n\nThe comparison therefore involves more than “vector retrieval versus grep.” Semantic search can also read files and inspect directories. Deep's distinguishing features are planning and subagent delegation.\n\n### Models, Data, and Scoring\n\n**Four models** are tested: Gemini 2.5 Flash, Gemini 2.5 Pro, Gemini 3 Flash, and an open-weight model from another provider, Qwen3-235B. Four models crossed with two architectures yield eight conditions. Temperature is zero, with a 900-second and 80-step limit per question. Exceeding either counts as failure, without a retry.\n\n**The dataset** is SWE-QA, a repository-wide question-answering benchmark. The authors use 15 Python repositories with 48 questions each, totaling 720. Twelve repositories come from SWE-bench and three from the newer SWE-bench-Live, ranging from roughly 13,000 to over 800,000 lines. Questions are roughly balanced across What, Why, Where, and How.\n\n**The judge** is Claude Sonnet 4.6, from a different provider than every tested model. It scores five SWE-QA dimensions—correctness, completeness, relevance, clarity, and reasoning—combining them into a 0–100 score. Scores of at least 70 are Pass, 50 to below 70 are Partial, and below 50 are Fail.\n\nThe authors validate the judge with three software engineers, each with at least two years of experience and none a paper author. Blind to the conditions, they assess 800 answers. Inter-engineer agreement is Fleiss' κ = 0.81; majority human judgment versus the LLM judge gives Cohen's κ = 0.74 and a mean absolute score difference of 5.2 points. Agreement is slightly lower on Deep answers, κ = 0.69 versus 0.78 for Semantic, which the authors attribute to Deep's longer, more complex responses.\n\n## Discussion\n\n### Accuracy: Semantic Wins for Every Model\n\n| Model | Pass (S) | Pass (D) | Gap | Cost per pass (S) | Cost per pass (D) |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| Gemini 2.5 Flash | 48.4% | 42.8% | 5.6 | $0.213 | $0.436 |\n| Gemini 2.5 Pro | 54.2% | 44.2% | 10.0 | $1.048 | $1.526 |\n| Gemini 3 Flash | 89.3% | 39.7% | 49.6 | $0.179 | $0.858 |\n| Qwen3-235B | 68.8% | 58.2% | 10.6 | $0.012 | $0.291 |\n\n*Compiled from Tables 3 and 4 of the paper. S is Semantic, D is Deep, and gaps are percentage points. The last two columns give US dollars per correct answer, calculated from measured tokens using the paper's single price schedule.*\n\nSemantic has a higher Pass rate for all four models. Averaging their Pass rates:\n\n$$\n\\frac{48.4+54.2+89.3+68.8}{4}\\approx65.2\\%,\\qquad\n\\frac{42.8+44.2+39.7+58.2}{4}\\approx46.2\\%\n$$\n\nThat gives the abstract's 65.2% versus 46.2%. Fail rates are 21.9% versus 34.4%: Deep turns some passes into partial answers and also produces more outright failures.\n\nBut gap size varies greatly. Gemini 2.5 Flash differs by only 5.6 points; Gemini 3 Flash by 49.6. Paired tests remain significant after multiple-comparison correction for all four models, but Cliff's δ is only about 0.14 for the two Gemini 2.5 models and 0.24 for Qwen3-235B. Only Gemini 3 Flash's 0.76 is large. The authors describe this carefully: the direction is consistent, the gains on Gemini 2.5 are modest, and the gap is large only on Gemini 3 Flash.\n\nCould the Pass threshold of 70 happen to favor Semantic? The authors sweep thresholds from 50 to 90. Semantic leads for every model at every threshold, with a minimum gap of 3.8 percentage points.\n\n### Cost: More Spending Does Not Buy Higher Accuracy\n\nDeep uses more input tokens for every model. The largest gap is Qwen3-235B: about 34,000 tokens per question for Semantic versus 761,000 for Deep, a 22-fold difference.\n\nAggregating across the four models, cost per correct answer is $0.32 versus $0.74. Deep is about 2.3 times as expensive.\n\nThe authors also examine per-question scores against spending. For input tokens and tool calls alike, fitted slopes are flat or negative under both architectures: **spending more does not produce higher scores**. They caution that this is correlational; harder questions naturally require more exploration and are more likely to be wrong.\n\nLatency has no consistent direction. Deep is faster for the two Gemini 2.5 models; Semantic is faster for Gemini 3 Flash and Qwen3-235B. The authors consider latency too sensitive to execution conditions for a general conclusion.\n\nDoes Deep's extra cost rise with repository size? No significant relationship appears across the 15 repositories, but the authors acknowledge the small sample and leave the question open.\n\n### Most of Deep's Work Happens Outside the Main Agent's View\n\nIncluding subagent actions, most Deep tool calls are ordinary file reads (41.6%) and text searches (29.1%), followed by to-do writing, directory listing, and delegation.\n\nBut a single delegation opens an entire subagent exploration. Of 2,878 Deep runs, 2,395 delegate at least once. Each run averages 2.1 delegations, and each subagent makes about ten calls. In total, **68.8% of tool calls happen inside subagents**, beyond the main agent's direct view.\n\nGemini 3 Flash illustrates this clearly: Semantic averages 8.6 tool calls per question, versus 44.6 for Deep.\n\n### Failure Analysis: Handoffs Are the Weak Spot\n\nThe authors classify all Fail traces along two independent dimensions: **failure mechanism**, or why it failed, and **answer symptom**, or how the answer is wrong. Claude Sonnet 4.6 performs the classification, validated by two authors manually labeling 400 examples. Agreement between those authors is κ = 0.78 for mechanisms and 0.86 for symptoms; their consensus agrees with automated classification at 0.82.\n\n[![Failure mechanisms by architecture: Semantic is dominated by retrieval or localization failures, while Deep Agentic is dominated by coordination breakdown](/content/papers/assets/61ecd2fcf8409a060dc66b99123841b7e1bceff328c5f1ee011397bb73773c74.png)](/content/papers/assets/61ecd2fcf8409a060dc66b99123841b7e1bceff328c5f1ee011397bb73773c74.png)\n\n*Fig. 19 from the paper, excerpted from [**page 27 of the v1 PDF**](https://arxiv.org/pdf/2608.01507v1#page=27). Blue on the left is Semantic; orange on the right is Deep Agentic. Numbers are percentages of each architecture's failures.*\n\nThere are 1,621 failures: 631 Semantic and 990 Deep. Their failure patterns differ sharply.\n\n**Semantic mostly fails to find things.** The largest category, retrieval or localization failure at the lower left, is 53.6%: despite an average of 12.6 tool calls, it never finds the correct file. Another 18.1% answer confidently after reading too little, and 17.1% find the right file but misunderstand it.\n\n**Deep mostly fails at handoffs.** The top orange bar, **coordination breakdown**, accounts for 41.8%. Handoffs between main agent and subagent fail to produce or integrate usable findings. This failure requires a subagent architecture, so Semantic has zero cases.\n\nWorse, most coordination breakdowns are **silent**. In 91% of them, the agent still produces a fluent, confident answer with no visible sign of trouble.\n\nAnother 13.5% of Deep failures involve loops. Overall, 24.2% of its failures produce no answer at all, versus 12.7% for Semantic.\n\nThe authors conclude that Deep Agentic Search adds **a new kind of failure**, one particularly difficult to detect from the output, while leaving other failures in place.\n\n### The Authors' Own Boundaries\n\nThe paper is careful about its scope.\n\n**First, every question is independent, read-only, and single-turn.** Subagents' real value may be keeping the main context clean across long sessions with multiple tasks. Every question here starts a new conversation, leaving no opportunity for that benefit to improve accuracy. The authors explicitly say their design cannot determine whether the balance changes in long sessions.\n\n**Second, Semantic assumes indexing is possible beforehand.** File systems that are too large, too volatile, or too irregular may require live exploration. An index is a snapshot: branches keep changing, so it must be updated continuously or answers can become subtly stale. The authors report that embedding all 15 repositories cost under one dollar; the real cost lies in detecting changes, invalidation, and rebuilding. A fixed snapshot, as used here, is especially favorable to indexing.\n\n**Third, each paradigm uses only one harness.** System prompts, subagent summaries, and step budgets all affect results. The authors do not compare other Deep Agent frameworks or claim that the gap's magnitude is fixed across implementations.\n\nThe evaluation also covers only SWE-QA, Python, and four models. These popular repositories may well have appeared in training data.\n\n## Conclusion\n\nFor this paper's setting—**fixed Python repositories that can be indexed, with one read-only question at a time**:\n\n1. **Semantic search is more accurate.** It has a higher Pass rate for every model, averaging 65.2% versus 46.2%, with gaps from 5.6 to 49.6 percentage points.\n2. **Semantic search is cheaper.** Cost per correct answer averages $0.32 versus $0.74. Deep's additional tokens do not buy higher scores.\n3. **Deep's main failure is the handoff.** Coordination between main agent and subagent accounts for 41.8% of failures, and 91% of those still produce fluent, confident wrong answers.\n\nThe authors' practical recommendation is to default to an indexed retrieval agent for read-only QA when a repository can be indexed. Reserve heavier subagent orchestration for needs outside single-question accuracy: clean contexts over long sessions, environments that cannot be indexed, or tasks requiring code writing and execution. A reasonable next step is a router: use inexpensive semantic indexing ordinarily and delegate exploration when needed.\n\nHanding off the work does keep the main agent's context clean. But it also has no way to tell whether the answer it gets back is right.\n","imageSizes":{"/content/papers/assets/44afeaa4b2a8e82e998b7d93f70c43ca555a0c6588ac386081af2aedc5612536.png":{"width":1258,"height":1083},"/content/papers/assets/61ecd2fcf8409a060dc66b99123841b7e1bceff328c5f1ee011397bb73773c74.png":{"width":944,"height":427}}}