{"notes":[{"id":"retrieval-augmented-generation/cag","category":"retrieval-augmented-generation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"CAG","subtitle":"Memorize More, Score More","author":"Z. Yuan","paperDate":"2024-12-20","publishedOn":"2026-07-19","paperTitle":"Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks","paperUrl":"https://arxiv.org/abs/2412.15605","bibliography":{"title":"Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks","authors":["Brian J Chan","Chao-Ting Chen","Jui-Hung Cheng","Hen-Hsen Huang"],"url":"https://arxiv.org/abs/2412.15605","pdfUrl":"https://arxiv.org/pdf/2412.15605v2","codeUrl":"https://github.com/hhhuang/CAG","preprint":{"id":"2412.15605","submittedOn":"2024-12-20","url":"https://arxiv.org/abs/2412.15605"},"publication":{"venue":"WWW Companion","year":2025,"kind":"conference","url":"https://doi.org/10.1145/3701716.3715490"},"citation":"@misc{retrievalaugmentedgenerationcag2024,\n  title = {{Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks}},\n  author = {Brian J Chan and Chao-Ting Chen and Jui-Hung Cheng and Hen-Hsen Huang},\n  year = {2024},\n  eprint = {2412.15605},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.15605}\n}\n","citationPath":"/content/papers/citations/retrieval-augmented-generation/cag.52cd8e1fb14c.bib"},"thumbnail":{"src":"/content/papers/assets/1ccb714a2b9e40fe07cb712c508763c4447c40b66c78a8c364a030957c65a789.png","width":694,"height":575},"locale":"en","path":"/content/papers/en/retrieval-augmented-generation/cag.422cfe775728.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"what-remains-after-the-model-reads","text":"What Remains After the Model Reads?","depth":3},{"id":"three-steps","text":"Three Steps","depth":3},{"id":"calculating-the-actual-size","text":"Calculating the Actual Size","depth":3},{"id":"why-put-everything-in-together","text":"Why Put Everything In Together?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"how-are-the-experiments-set-up","text":"How Are the Experiments Set Up?","depth":3},{"id":"accuracy-mostly-ahead-with-exceptions","text":"Accuracy: Mostly Ahead, with Exceptions","depth":3},{"id":"speed-the-saving-comes-from-avoiding-rereading","text":"Speed: The Saving Comes from Avoiding Rereading","depth":3},{"id":"questions-this-paper-leaves-open","text":"Questions This Paper Leaves Open","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"This paper has a title with attitude: “**Don't Do RAG**.”\n\nRAG—Retrieval-Augmented Generation—has become the standard way to connect language models to external information. A user asks a question; the system finds a few relevant passages in a knowledge base, puts them before the question, and asks the model to answer from them. Internal documents, product FAQs, and customer-support records often reach chatbots this way.\n\nThe workflow is useful, but things can go wrong at every stage.\n\nIf the retriever selects the wrong passage, the model has the wrong material to work with. If it finds the right material but only half of it, the answer is incomplete. Splitting documents, building indexes, tuning top-k, and choosing an embedding model all add maintenance work.\n\nBy 2024, open-source models had context windows exceeding a hundred thousand tokens. The authors therefore ask a direct question:\n\n- **If the entire knowledge base already fits, why retrieve first?**\n\n\n> [!TIP]\n> \n> **Who Are the Authors?**\n> \n> The three co-first authors, Brian J Chan, Chao-Ting Chen, and Jui-Hung Cheng, are from the Department of Computer Science at National Chengchi University. Corresponding author Hen-Hsen Huang is from Academia Sinica's Institute of Information Science. The paper appeared at WWW Companion '25; this note follows arXiv v2. Code is available on [**GitHub**](https://github.com/hhhuang/CAG).\n\n\n## Defining the Problem\n\nPutting an entire knowledge base into a prompt is not particularly new. The problem is its cost.\n\nSuppose our knowledge base contains 85,000 tokens. Every question sends those 85,000 tokens plus the question to the model. The model must read from the first token through the entire reference before it can begin writing the answer.\n\nThe next question arrives. The information has not changed at all, but the model reads it from the beginning again.\n\nThis prompt-reading stage is called **prefill**. Longer prompts take longer to prefill, increasing the user's wait for the first output token: **TTFT**, or time to first token.\n\nThe actual question the authors want to address is:\n\n- **If the information stays unchanged, can we read it just once?**\n\nTo answer that, we need to know what the model retains after “reading.”\n\n## Solving the Problem\n\n### What Remains After the Model Reads?\n\nMost current language models are decoder-only Transformers. Self-attention at each layer projects every token into Query, Key, and Value vectors.\n\nWhen generating the next token, its new Query is compared with **the Keys of all preceding tokens**, then used to weight **the Values of all preceding tokens**.\n\nThe crucial point is that those earlier Keys and Values stay unchanged once computed.\n\nImplementations therefore save them for the next step. This is the **KV cache**, already used during ordinary token-by-token generation to avoid repeated computation.\n\n\n> [!TIP]\n> \n> **Why Don't Earlier Keys and Values Change?**\n> \n> The decoder uses a causal mask: each token can see only itself and preceding tokens. Appending new text leaves the earlier tokens' computations unchanged.\n> \n> Conversely, changing something earlier in the text requires recomputing the Keys and Values of all subsequent tokens. We will return to that limitation later.\n\n\nCAG—Cache-Augmented Generation—extends this reuse from “within one question” to “across all questions”:\n\n- **Read the entire knowledge base once and save its KV cache. Every later question only needs to continue from there.**\n\nThe paper compares the two workflows below. Pink marks components used at inference time:\n\n[![RAG versus CAG: RAG retrieves knowledge through an IR Model at inference time; CAG preloads knowledge into a Context Cache offline and processes only the question at inference](/content/papers/assets/1ccb714a2b9e40fe07cb712c508763c4447c40b66c78a8c364a030957c65a789.png)](/content/papers/assets/1ccb714a2b9e40fe07cb712c508763c4447c40b66c78a8c364a030957c65a789.png)\n\n*Figure 1 from the paper, excerpted from [**page 2 of the v2 PDF**](https://arxiv.org/pdf/2412.15605v2#page=2). RAG is above and CAG below; pink components run during online inference.*\n\nIn the upper RAG workflow, every question goes through an IR Model to retrieve information. The resulting Context and question $q$ are then passed to the LLM.\n\nIn the lower CAG workflow, “Offline Preloading” turns the knowledge base into a Context Cache. Only question $q$ is pink at inference time. The retriever has disappeared entirely.\n\n### Three Steps\n\nThe authors describe three steps.\n\n**Step 1: Preload knowledge.** Organize the document collection $\\mathcal{D}=\\{d_1,d_2,\\dots\\}$ and let the model read it once, producing a KV cache:\n\n$$\nC_{KV}=\\text{KV-Encode}(\\mathcal{D})\n$$\n\nThe cache can remain in memory or be saved to disk. The document-reading cost is paid once, regardless of how many questions follow.\n\n**Step 2: Inference.** Load $C_{KV}$ and append the user's question $q$:\n\n$$\nr=\\mathcal{M}(\\mathcal{D}\\oplus q)=\\mathcal{M}(q\\mid C_{KV})\n$$\n\nThe left side concatenates the documents and question for the model. The right side gives the model its cache and reads only the question. The computation gives the same result; the document portion simply avoids being reread.\n\n**Step 3: Reset the cache.** Answering the first question appends question and answer tokens to the KV cache. Remove those additions before the second question, or it will see the first conversation.\n\nSince the cache is append-only, remember its length after reading the knowledge base and truncate everything after that position. There is no need to reload it from disk.\n\n### Calculating the Actual Size\n\n“Save the KV cache” sounds abstract. Let's calculate what it looks like for the paper's Llama 3.1 8B model.\n\nAccording to Table 3 of the [**Llama 3 paper**](https://arxiv.org/abs/2407.21783), the 8B model has 32 layers, hidden dimension 4096, and 32 attention heads, but only eight Key/Value heads through grouped-query attention. Each head has $4096/32=128$ dimensions.\n\nAt each layer, the Key tensor stored for one token therefore has shape:\n\n$$\n[\\text{batch},\\ \\text{kv\\_heads},\\ \\text{seq\\_len},\\ \\text{head\\_dim}]=[1,\\ 8,\\ 1,\\ 128]\n$$\n\nThe Value has the same shape. Across all layers, one token requires this many stored numbers:\n\n$$\n2\\ (\\text{K and V})\\times 32\\ (\\text{layers})\\times 8\\ (\\text{head})\\times 128\\ (\\text{dimensions})=65{,}536\n$$\n\nAt 16 bits, or two bytes, per number, that is 128 KiB per token.\n\nThe largest HotPotQA setting contains about 85k tokens:\n\n$$\n85{,}000\\times 128\\ \\text{KiB}\\approx 11.1\\ \\text{GB}\n$$\n\nThe model weights themselves occupy about 16 GB at 16-bit precision. The cache is already almost 70% of that.\n\n\n> [!NOTE]\n> \n> This is an estimate from the model configuration, assuming a 16-bit cache. The paper reports neither actual cache size nor memory usage. The official code loads the model with `torch.float16` by default and also offers a 4-bit quantization option.\n\n\nWith the tensor shape in mind, the reset step in the official code is easy to understand:\n\n```python\ndef clean_up(kv: DynamicCache, origin_len: int):\n    \"\"\"\n    Truncate the KV Cache to the original length.\n    \"\"\"\n    for i in range(len(kv.key_cache)):\n        kv.key_cache[i] = kv.key_cache[i][:, :, :origin_len, :]\n        kv.value_cache[i] = kv.value_cache[i][:, :, :origin_len, :]\n```\n\n*From `kvcache.py` in [**hhhuang/CAG**](https://github.com/hhhuang/CAG/blob/5c0d8ed6e28f34c72bb7208068f55cb3147d0c8c/kvcache.py#L120-L126), MIT license.*\n\n`kv.key_cache[i]` is the Key tensor at layer $i$, with shape `[1, 8, seq_len, 128]`. The slice `[:, :, :origin_len, :]` operates along the third dimension, sequence length, retaining only the knowledge-base portion.\n\n`origin_len` is recorded after preloading. Before each later question, the cache is cut back to that length. The entire “reset” consists of two slices per layer across 32 layers.\n\n### Why Put Everything In Together?\n\nAt this point you might ask: if caching works so well, why not cache **each document separately**, then concatenate the caches of whichever documents are retrieved?\n\nThe problem is in the earlier callout. A token's Key and Value depend on all preceding text and on its position in the sequence. When Document B is computed alone, it knows nothing of Document A before it, and its positions start from the beginning. Joining those caches directly differs from actually reading A followed by B.\n\nThe introduction mentions TurboRAG, which precomputes passage-level KV caches and needs additional position-ID rearrangement to operate properly. CAG takes a blunter approach: **if the knowledge base is fixed, read the whole thing together**. Both retrieval and the cache-joining problem disappear.\n\nThe price is that changing any text in the knowledge base requires recomputing the cache after that position.\n\n\n> [!TIP]\n> \n> **Isn't This Just Prompt Caching?**\n> \n> Someone on GitHub [**asked a similar question**](https://github.com/hhhuang/CAG/issues/18): wouldn't placing the entire document in the system prompt give the same result?\n> \n> Contributor brian030128 replied that the output is indeed the same. The difference is that converting a prompt into a KV cache is computationally expensive; CAG does that ahead of time, saving TTFT. Others subsequently asked how this differs from API providers' context caching.\n> \n> Mechanically, they are very close. [**Claude's prompt caching**](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) and [**Gemini's context caching**](https://ai.google.dev/gemini-api/docs/caching) can both reuse repeated long prompts. This paper's contribution is more about comparing full-knowledge-base preloading and caching against RAG on the same QA tasks.\n\n\n## Discussion\n\n### How Are the Experiments Set Up?\n\nThe authors use two QA datasets:\n\n- **SQuAD 1.0**: Answers are spans in a passage, leaning toward locating information in one passage.\n- **HotPotQA**: Answers combine multiple documents and require multi-hop reasoning.\n\nTo study growing knowledge bases, they create small, medium, and large subsets of each dataset. For HotPotQA small, for example, they sample 16 documents, concatenate them into a long reference, and select only questions related to those documents.\n\n| Dataset | Size | Documents | Tokens | Questions |\n| --- | --- | ---: | ---: | ---: |\n| HotPotQA | Small | 16 | 21k | 1,392 |\n| HotPotQA | Medium | 32 | 43k | 1,056 |\n| HotPotQA | Large | 64 | 85k | 1,344 |\n| SQuAD | Small | 3 | 21k | 500 |\n| SQuAD | Medium | 4 | 32k | 500 |\n| SQuAD | Large | 7 | 50k | 500 |\n\n*Compiled from Table 1 of the paper.*\n\nAll systems answer with Llama 3.1 8B, running on eight Tesla V100 32GB GPUs. The comparisons are two RAG implementations built with LlamaIndex:\n\n- **Sparse RAG**: BM25, ranking through keyword matching and term frequency.\n- **Dense RAG**: An OpenAI embedding index, ranking through semantic similarity.\n\nBoth try retrieving the top 1, 3, 5, and 10 passages.\n\nScoring uses **BERTScore**, measuring semantic similarity between the model's answer and the reference answer.\n\n\n> [!TIP]\n> \n> **What Is BERTScore?**\n> \n> It passes both texts through a BERT-like model, extracts token vectors, and matches them to calculate similarity. An advantage is that “Paris” and “Paris, the capital of France” are not treated as completely different strings.\n> \n> But it measures similarity rather than correctness. Two answers differing by one number can still have a high BERTScore. Interpret differences in the third decimal place cautiously in the table below.\n\n\n### Accuracy: Mostly Ahead, with Exceptions\n\n[![Table 2: BERTScore for Sparse RAG and Dense RAG at top-1, 3, 5, and 10, and CAG across small, medium, and large HotPotQA and SQuAD settings](/content/papers/assets/13f5f729ad0290d70a0afcceaf6ec535895cb1017a05906e8dcfa0a9e91cae31.png)](/content/papers/assets/13f5f729ad0290d70a0afcceaf6ec535895cb1017a05906e8dcfa0a9e91cae31.png)\n\n*Table 2 from the paper, excerpted from [**page 4 of the v2 PDF**](https://arxiv.org/pdf/2412.15605v2#page=4). HotPotQA is on the left, SQuAD on the right; bold marks the highest score in each setting.*\n\nStart with the “CAG (Ours)” row at the bottom of each group.\n\nCAG scores highest on both datasets in the Small and Medium settings. For HotPotQA small, for example, CAG gets 0.7951, while the best RAG result is Sparse top-5 at 0.7676.\n\nNow look at Large. CAG's 0.7734 still leads on SQuAD. On HotPotQA, however, the bold score belongs to Sparse RAG top-5 at **0.7535**. CAG gets only 0.7407, below Sparse top-3's 0.7463 and nearly identical to Dense top-3's 0.7409.\n\nThe authors also note in the prose that CAG's advantage shrinks as the data grows, citing research on degradation with very long inputs.\n\nAnother detail deserves attention. The introduction cites RULER and gives Llama 3.1 8B an **effective context length of 32K**. HotPotQA large, at about 85k tokens, is well beyond that. This might help explain CAG's loss there, although the paper does not isolate the cause; it is our interpretation of the table.\n\nOne more pattern: BM25 is the highest-scoring RAG configuration in five of the six settings. Only SQuAD medium gives Dense top-10 a narrow 0.0009 win. The authors suggest the datasets may be easy enough that keyword matching finds most of the required information. In this comparison, keyword retrieval was sufficient for most of the information needed.\n\n### Speed: The Saving Comes from Avoiding Rereading\n\nNow consider time.\n\n[![Table 3: retrieval and generation times for RAG configurations, CAG, and In-Context Learning across the three HotPotQA sizes](/content/papers/assets/fbdd7c78239b53372fb8d168a05e1cb9ffda9cfdc3c6301abb36dd2913389490.png)](/content/papers/assets/fbdd7c78239b53372fb8d168a05e1cb9ffda9cfdc3c6301abb36dd2913389490.png)\n\n*Table 3 from the paper, excerpted from [**page 4 of the v2 PDF**](https://arxiv.org/pdf/2412.15605v2#page=4). Units are seconds. Retrieval is retrieval time; Generation is generation time.*\n\nThere is an additional comparison: **In-Context Learning**. It sends the full reference and question directly to the model and computes the KV cache anew for every question—the “read everything from the beginning” approach discussed earlier.\n\nComparing this row with CAG isolates what preloading saves:\n\n| HotPotQA size | Reread every question | CAG | Speedup |\n| --- | ---: | ---: | ---: |\n| Small (21k) | 9.32 s | 0.85 s | About 11× |\n| Medium (43k) | 26.37 s | 1.41 s | About 19× |\n| Large (85k) | 92.08 s | 2.26 s | About 41× |\n\nThe larger the knowledge base, the more expensive rereading becomes, and the greater the benefit of preloading.\n\nCompare CAG with Sparse RAG, though, and the picture changes. In the Large setting, BM25 top-3 takes only 0.0008 seconds for retrieval and 0.67 seconds for generation, under one second in total. CAG takes 2.26 seconds. BM25 gives the model just three short passages, so the context it processes is already short.\n\nCAG's own time also increases with knowledge-base size: 0.85, 1.41, and 2.26 seconds. Caching avoids rereading, but attention still scans the entire 85k-token cache for every generated token.\n\nOne number is missing from the table: **how long the cache takes to build**. Table 3 counts only per-question inference after preloading. The official code records preload time, but the paper does not report it. For a knowledge base updated daily, that number matters.\n\n### Questions This Paper Leaves Open\n\nThe Limitations section is direct: all relevant documents must fit into the model's context. The method suits bounded settings such as a small company's internal knowledge base, FAQs, or a support center, and becomes impractical for much larger datasets.\n\nThe earlier derivation also suggests several engineering concerns:\n\n1. **Frequent document updates.** The cache after the edited position must be recomputed. The paper does not measure update costs.\n2. **Different users see different documents.** One cache contains the entire knowledge base; different permissions may require separate caches for different document combinations. This is our inference, not a topic discussed in the paper.\n3. **The cache is tied to the model.** A KV cache is an intermediate state computed by a particular model version. Changing models requires rebuilding it.\n\n## Conclusion\n\nCAG's proposal fits in one sentence: **when a fixed knowledge base fits in the context window, read it once, save the KV cache, and read only the question thereafter.**\n\nIn the paper's experimental setting:\n\n- **It is much faster than rereading everything for each question.** HotPotQA large falls from 92.08 to 2.26 seconds, about a 41× speedup.\n- **Accuracy is mostly comparable to or better than RAG.** CAG leads in both Small and Medium settings; in Large, it still leads on SQuAD but loses to BM25 top-5 on HotPotQA.\n- **Lightweight BM25 can be faster than CAG.** RAG using just three short passages takes under one second in total.\n\nThese results come from a single 8B model, knowledge bases of at most about 85k tokens, and BERTScore as a semantic-similarity metric.\n\nThe title says “Don't Do RAG.” The paper offers a gentler answer: if a knowledge base is small, stable, and queried repeatedly, consider skipping retrieval and memorizing it first.\n\nIf it is too big to memorize, you will still have to look things up.\n","imageSizes":{"/content/papers/assets/1ccb714a2b9e40fe07cb712c508763c4447c40b66c78a8c364a030957c65a789.png":{"width":694,"height":575},"/content/papers/assets/13f5f729ad0290d70a0afcceaf6ec535895cb1017a05906e8dcfa0a9e91cae31.png":{"width":694,"height":1028},"/content/papers/assets/fbdd7c78239b53372fb8d168a05e1cb9ffda9cfdc3c6301abb36dd2913389490.png":{"width":689,"height":717}}}