{"notes":[{"id":"model-evaluation/nolima","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NoLiMa","subtitle":"Reading Between the Lines","author":"Z. Yuan","paperDate":"2025-02-07","publishedOn":"2026-07-30","paperTitle":"NoLiMa: Long-Context Evaluation Beyond Literal Matching","paperUrl":"https://arxiv.org/abs/2502.05167","bibliography":{"title":"NoLiMa: Long-Context Evaluation Beyond Literal Matching","authors":["Ali Modarressi","Hanieh Deilamsalehy","Franck Dernoncourt","Trung Bui","Ryan Rossi","Seunghyun Yoon","Hinrich Schütze"],"url":"https://arxiv.org/abs/2502.05167","pdfUrl":"https://arxiv.org/pdf/2502.05167v3","codeUrl":"https://github.com/adobe-research/NoLiMa","preprint":{"id":"2502.05167","submittedOn":"2025-02-07","url":"https://arxiv.org/abs/2502.05167"},"publication":{"venue":"ICML","year":2025,"kind":"conference","url":"https://proceedings.mlr.press/v267/modarressi25a.html"},"citation":"@misc{modelevaluationnolima2025,\n  title = {{NoLiMa: Long-Context Evaluation Beyond Literal Matching}},\n  author = {Ali Modarressi and Hanieh Deilamsalehy and Franck Dernoncourt and Trung Bui and Ryan Rossi and Seunghyun Yoon and Hinrich Schütze},\n  year = {2025},\n  eprint = {2502.05167},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.05167}\n}\n","citationPath":"/content/papers/citations/model-evaluation/nolima.264a4512e69f.bib"},"thumbnail":{"src":"/content/papers/assets/d3917196027c3729a255d140f88c8ddc4cf99d5ea087de73ead2f387a8befb47.png","width":1440,"height":919},"locale":"en","path":"/content/papers/en/model-evaluation/nolima.8d1212b266fc.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"a-needle-that-cannot-be-found-by-matching-words","text":"A Needle That Cannot Be Found by Matching Words","depth":3},{"id":"cleaning-the-hay","text":"Cleaning the Hay","depth":3},{"id":"how-is-it-scored","text":"How Is It Scored?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"main-results","text":"Main Results","depth":3},{"id":"two-hops-are-harder-than-one-inverted-order-is-harder-than-default","text":"Two Hops Are Harder Than One; Inverted Order Is Harder Than Default","depth":3},{"id":"is-the-problem-position-or-length","text":"Is the Problem Position or Length?","depth":3},{"id":"does-thinking-help","text":"Does Thinking Help?","depth":3},{"id":"put-the-literal-clues-back","text":"Put the Literal Clues Back","depth":3},{"id":"what-about-newer-models","text":"What About Newer Models?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-evaluation/lost-in-the-middle","relation":"referenced","topic":""}]},{"id":"model-evaluation/lost-in-the-middle","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Lost in the Middle","subtitle":"The Curse of the U","author":"Z. Yuan","paperDate":"2023-07-06","publishedOn":"2026-06-09","world":"external/lost-in-the-middle","paperTitle":"Lost in the Middle: How Language Models Use Long Contexts","paperUrl":"https://arxiv.org/abs/2307.03172","bibliography":{"title":"Lost in the Middle: How Language Models Use Long Contexts","authors":["Nelson F. Liu","Kevin Lin","John Hewitt","Ashwin Paranjape","Michele Bevilacqua","Fabio Petroni","Percy Liang"],"url":"https://arxiv.org/abs/2307.03172","pdfUrl":"https://arxiv.org/pdf/2307.03172v3","codeUrl":null,"preprint":{"id":"2307.03172","submittedOn":"2023-07-06","url":"https://arxiv.org/abs/2307.03172"},"publication":{"venue":"Transactions of the Association for Computational Linguistics","year":2024,"kind":"journal","url":"https://aclanthology.org/2024.tacl-1.9/"},"citation":"@misc{modelevaluationlostinthemiddle2023,\n  title = {{Lost in the Middle: How Language Models Use Long Contexts}},\n  author = {Nelson F. Liu and Kevin Lin and John Hewitt and Ashwin Paranjape and Michele Bevilacqua and Fabio Petroni and Percy Liang},\n  year = {2023},\n  eprint = {2307.03172},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2307.03172}\n}\n","citationPath":"/content/papers/citations/model-evaluation/lost-in-the-middle.77681ac150c1.bib"},"thumbnail":{"src":"/content/papers/assets/14c392ed0cd20cff36405d2cccc1e1d2e317e1c84de40c4eae37cde79f4a31e3.png","width":1278,"height":425},"locale":"en","path":"/content/papers/en/model-evaluation/lost-in-the-middle.27abe25573b3.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"multi-document-question-answering","text":"Multi-Document Question Answering","depth":3},{"id":"keyvalue-retrieval","text":"Key–Value Retrieval","depth":3},{"id":"models-tested","text":"Models Tested","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-u-shaped-curve","text":"The U-Shaped Curve","depth":3},{"id":"are-extended-models-better-at-using-long-contexts","text":"Are Extended Models Better at Using Long Contexts?","depth":3},{"id":"even-copying-a-string-can-go-wrong","text":"Even Copying a String Can Go Wrong","depth":3},{"id":"why-does-this-happen","text":"Why Does This Happen?","depth":3},{"id":"what-about-adding-more-documents","text":"What About Adding More Documents?","depth":3},{"id":"what-happened-next","text":"What Happened Next?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-evaluation/nolima","relation":"citing","topic":""}]}],"text":"From 2024 onward, model launches advertised ever larger context windows: 128K, 1M, even 2M tokens.\n\nThe most common way to demonstrate that a model can read that much is the **needle-in-a-haystack** test (NIAH): hide a sentence inside a large amount of irrelevant text, then ask the model what it says.\n\nThe popular version Greg Kamradt introduced in 2023 hid this sentence:\n\n> The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.\n\nThe question was:\n\n> What is the most fun thing to do in San Francisco?\n\nSee the problem?\n\n“San Francisco” appears unchanged in both the question and the needle. The model can find the same words in the context, with the answer right beside them, without understanding anything.\n\nThis paper from LMU Munich and Adobe Research, published at ICML 2025, asks: **if we remove those literal clues, can the model still find the needle?**\n\n## Defining the Problem\n\nThe authors begin by measuring something: in existing long-context benchmarks, how similar are the words in the question and the passage containing the answer?\n\nThey use ROUGE precision to measure how many words from the question appear in the relevant passage. Higher values mean more shared words:\n\n| Benchmark | ROUGE-1 | ROUGE-2 | ROUGE-L |\n| --- | ---: | ---: | ---: |\n| Vanilla NIAH | 0.905 | 0.789 | 0.855 |\n| RULER S-NIAH | 0.571 | 0.461 | 0.500 |\n| RULER QA | 0.809 | 0.437 | 0.693 |\n| ∞Bench QA | 0.966 | 0.545 | 0.960 |\n| **NoLiMa** | **0.069** | **0.002** | **0.067** |\n\n*Compiled from Table 1 of the paper.*\n\nVanilla NIAH reaches a ROUGE-1 of 0.905: about 90% of the question's words can be found in the needle. Multi-document and long-document QA benchmarks also have substantial overlap.\n\nThe authors argue that attention is good at recalling repeated patterns. This overlap may make finding information in a long context much easier. How much of the advertised length is supported by simple word matching?\n\nThey therefore set themselves a goal: build a benchmark where **the task itself is simple, but literal matching provides no help**.\n\n## Solving the Problem\n\n### A Needle That Cannot Be Found by Matching Words\n\nNoLiMa has the same skeleton as NIAH: one needle in a pile of hay. The difference is how the needle and question are written.\n\nHere is a needle:\n\n> Actually, Yuki lives next to the Semper Opera House.\n\nIts question is:\n\n> Which character has been to Dresden?\n\nThe question contains neither Yuki nor Semper Opera House; the needle contains no Dresden. The only connection is a fact about the world: **the Semper Opera House is in Dresden**.\n\nThe model must make that association to connect the question with the needle and answer “Yuki.”\n\nThe authors distinguish two keywords: $W_q$ in the question, here Dresden, and $W_n$ in the needle, here Semper Opera House.\n\nAdd another layer and it becomes **two-hop**:\n\n> Which character has been to the state of Saxony?\n\nNow the model must reason through two connections: the opera house is in Dresden, and Dresden is in Saxony.\n\nSome needles use commonsense rather than world knowledge. For example, the needle says, “Then Yuki mentioned that he has been vegan for years,” and the question asks which character cannot eat fish dishes.\n\n\n> [!TIP]\n> \n> **Principles for Designing Needles and Questions**\n> \n> - **Keep the association simple.** Without hay, the model should answer easily.\n> - **Randomize names.** Sample from a diverse name pool to reduce tokenization issues and demographic bias. Exclude names already present in the hay.\n> - **Add a short lead-in.** A phrase such as “Actually,” naturally separates the needle from the preceding text.\n> \n> Each needle also has two word orders: the **default** puts the name first (“Yuki lives next to the Semper Opera House”); the **inverted** version puts the keyword first (“Next to the Semper Opera House lives Yuki”). There are five sets containing 58 question–needle pairs in total.\n\n\n### Cleaning the Hay\n\nThe hay comes from ten openly licensed books. To reduce the possibility of memorization, the authors do not use whole books directly. They concatenate randomly selected passages of fewer than 250 tokens from randomly selected books.\n\nThen comes an important cleanup. Two kinds of content must be removed:\n\n1. **Words too similar to question keywords.** Use Contriever to embed each word in the hay, find the 20 most similar words for each $W_q$, inspect them manually, and remove sentences containing those words.\n2. **Content that could serve as an answer.** Split the hay into passages of about 250 tokens and give each passage and question to Llama 3.3 70B to check for answers. The authors manually review flagged passages for removal, repeating until no further removals are needed.\n\nOtherwise, a character in the hay might happen to have visited Dresden, making scoring ambiguous.\n\n### How Is It Scored?\n\nEach needle is placed at 26 evenly spaced positions within the tested length. With five random haystacks and 58 question–needle pairs, that gives 7,540 tests per length. Lengths range from 250, 500, and 1K up to 32K tokens.\n\nEvery question asks for a character's name, so an answer is correct if it contains the right name.\n\nTwo metrics need clarification.\n\n**Base score** uses the three shortest lengths: 250, 500, and 1K. For each question–needle pair, average over the five haystacks, take the highest of the three length scores, then average across questions. It describes how well the model can answer with almost no hay interference.\n\n**Effective length** is the largest tested length at which performance remains at least 85% of the base score.\n\nLet's calculate this for GPT-4o. Its base score is 99.3, giving a threshold of:\n\n$$\n99.3\\times 0.85\\approx 84.4\n$$\n\nIts scores by length are:\n\n| Length | 1K | 2K | 4K | 8K | 16K | 32K |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: |\n| GPT-4o | 98.1 | 98.0 | 95.7 | 89.2 | 81.6 | 69.7 |\n\n*Excerpted from Table 3 of the paper.*\n\nAt 8K, 89.2 is still above 84.4. At 16K, 81.6 is below it. GPT-4o's effective length is therefore **8K**, against an advertised 128K.\n\nUsing each model's own base score separates intrinsic task difficulty from degradation caused by length. A model that struggles even in short contexts starts with a lower base score and a lower threshold.\n\n## Discussion\n\n### Main Results\n\nThe authors test 13 models advertised to support at least 128K tokens:\n\n| Model | Advertised length | Effective length | Base score | 32K |\n| --- | ---: | ---: | ---: | ---: |\n| GPT-4o | 128K | 8K | 99.3 | 69.7 |\n| Llama 3.3 70B | 128K | 2K | 97.3 | 42.7 |\n| Llama 3.1 405B | 128K | 2K | 94.7 | 38.0 |\n| Llama 3.1 70B | 128K | 2K | 94.5 | 43.2 |\n| Gemini 1.5 Pro | 2M | 2K | 92.6 | 48.2 |\n| Jamba 1.5 Mini | 256K | <1K | 92.4 | 43.6 |\n| Command R+ | 128K | <1K | 90.9 | 7.4 |\n| Gemini 2.0 Flash | 1M | 4K | 89.4 | 41.0 |\n| Mistral Large 2 | 128K | 2K | 87.9 | 18.8 |\n| Claude 3.5 Sonnet | 200K | 4K | 87.5 | 29.8 |\n| Gemini 1.5 Flash | 1M | <1K | 84.7 | 28.6 |\n| GPT-4o mini | 128K | <1K | 84.8 | 13.7 |\n| Llama 3.1 8B | 128K | 1K | 76.7 | 14.2 |\n\n*Compiled from Table 3 of the paper. The complete table also gives scores at lengths from 1K through 16K.*\n\nStart with the base-score column: most models score in the 80s or 90s in short contexts, so the needles themselves are not difficult.\n\nNow look at effective length. Apart from GPT-4o at 8K, all models are at 4K or below, many at 2K or even below 1K.\n\nPlotting advertised and effective lengths on the same logarithmic axis makes the gap clearer:\n\n[![Advertised context lengths versus NoLiMa effective lengths for 13 models; most advertise at least 128K, while effective lengths are mostly 2K or below](/content/papers/assets/08541b413f077ef8e8f464be627b19590979a35b8c741a89d532e688f2b22b21.png)](/content/papers/assets/08541b413f077ef8e8f464be627b19590979a35b8c741a89d532e688f2b22b21.png)\n\n*Drawn for this note using the numbers in Table 3; no original paper artwork is reused. Blue points show effective length, gray points advertised length, and hollow points an effective length below 1K. Click to enlarge.*\n\nFinally, look at 32K. Eleven of the 13 models score below half their own base score. Only GPT-4o and Gemini 1.5 Pro retain at least half.\n\nThe authors also compare the same model across benchmarks. Llama 3.1 70B has an effective length of 16K on BABILong and 32K on RULER. On NoLiMa, it falls to 2K.\n\nAnother interesting observation: the ranking by base score differs from the ranking by effective length. Claude 3.5 Sonnet and Gemini 2.0 Flash have lower base scores than Llama 3.3 70B, yet effective lengths of 4K rather than 2K. At 4K, their raw scores also exceed those of some models with higher base scores.\n\n### Two Hops Are Harder Than One; Inverted Order Is Harder Than Default\n\nThe authors examine two factors using GPT-4o and Llama 3.3 70B.\n\n**Hop count**: At the same length, two-hop questions are harder, and the gap widens as context grows. Within 4K, GPT-4o still handles both reasonably well.\n\n**Word order**: The inverted “Next to the Semper Opera House lives Yuki” is harder than the default “Yuki lives next to the Semper Opera House.”\n\nThe authors link this to causal attention in the decoder. With default order, the name appears first, so the representation at “Semper Opera House” already carries information about Yuki. Dresden in the question can connect to the opera house and then to Yuki. With inverted order, Yuki has not appeared when the opera house is read; connecting there yields no name, leaving a weaker signal on the name's side. The authors also state that a deeper mechanistic analysis is beyond the paper's scope.\n\n### Is the Problem Position or Length?\n\nReaders of [**Lost in the Middle**](https://arxiv.org/abs/2307.03172) may wonder whether scores decline because the needle happens to fall in the middle.\n\nThe authors perform two position sweeps with Llama 3.3 70B.\n\nThe first is a **full-range sweep**, placing the needle from 0% to 100% of the context. One-hop questions show the familiar pattern: the middle is worse. Two-hop questions behave differently: as context grows, the whole curve sinks, including its beginning and end.\n\nThe second design is clever: **align within the last 2K**. Whether the total context is 4K or 32K, place the needle at fixed positions within the final 2K tokens.\n\nThe needle-to-question distance is then identical across total lengths. Llama 3.x uses RoPE, a relative positional encoding based on distances between tokens. If positional encoding were the cause, the same distances should produce similar scores.\n\nInstead, the two-hop curves are fairly flat within the last 2K, with little position effect, but the entire curve falls as total context length grows.\n\nWith distance held fixed, the authors attribute the main limitation to length itself: there are more tokens to attend to, and without literal clues, attention has a harder time finding the relevant fact.\n\n### Does Thinking Help?\n\nIf association is required, what about asking the model to write its reasoning first—Chain-of-Thought, or CoT?\n\nThe authors compare direct answers with answers preceded by at most three reasoning sentences using Llama 3.3 70B:\n\n| Length | 4K | 8K | 16K | 32K |\n| --- | ---: | ---: | ---: | ---: |\n| One-hop, direct | 90.3 | 84.1 | 73.2 | 56.2 |\n| One-hop, CoT | 95.6 | 91.1 | 82.6 | 60.6 |\n| Two-hop, direct | 70.7 | 57.4 | 42.7 | 25.9 |\n| Two-hop, CoT | 82.4 | 70.1 | 56.7 | 34.3 |\n\n*Compiled from Table 4 of the paper.*\n\nCoT helps, especially on two-hop questions. But two-hop CoT only roughly catches up with one-hop direct answering, and performance remains poor at 16K and above.\n\nThe authors explain that these questions contain only one clue and cannot be decomposed into easier steps. The hard part is associating Dresden with the Semper Opera House, and that step is precisely what becomes difficult in a long context.\n\nThey also select the ten hardest question–needle pairs as NoLiMa-Hard to test reasoning models. GPT-o1, GPT-o3 Mini, and DeepSeek R1-Distill-Llama-70B all have near-perfect base scores, but all fall below 50% at 32K: 31.1 for GPT-o1, 18.9 for o3 Mini, and 20.7 for distilled R1.\n\n### Put the Literal Clues Back\n\nThe final ablations test the central hypothesis: what happens when literal clues return?\n\nThe authors create two comparisons:\n\n- **Direct**: The question explicitly mentions the needle's keyword, as in vanilla NIAH.\n- **Multiple Choice**: Keep the question, but add four name options: one from the needle and three from the hay. This introduces a literal link between question and needle—the name.\n\n| Setting | 8K | 16K | 32K |\n| --- | ---: | ---: | ---: |\n| Direct | 98.3 | 98.5 | 98.5 |\n| One-hop | 84.1 | 73.2 | 56.2 |\n| One-hop + choices | 98.7 | 97.4 | 93.1 |\n| Two-hop | 57.4 | 42.7 | 25.9 |\n| Two-hop + choices | 96.3 | 94.6 | 87.2 |\n\n*Compiled from Table 6 of the paper, using Llama 3.3 70B.*\n\nThe result is clear. Direct questions still score 98.5 at 32K. Adding answer choices raises two-hop performance at 32K from 25.9 to 87.2.\n\nLiteral clues can also mislead. The authors insert a sentence containing $W_q$, such as Dresden, but unrelated to the needle and question. Its wording makes it look more like an answer than the real needle. With this distractor, GPT-4o's effective length shrinks from 8K to 1K; Llama 3.3 70B fares even worse.\n\n### What About Newer Models?\n\nThe v3 appendix tests newer models and extends some evaluations to 64K and 128K. To control API costs, it reduces placement positions from 26 to 11 per length.\n\nGPT-4.1 advertises 1M, has an effective length of 16K, and still scores 64.7 at 128K. GPT-4o scores 56.0 at 128K; Gemini 2.0 Flash falls to 16.4. Llama 4 Scout advertises 10M but has an effective length of only 1K.\n\nA first-page footnote states the conclusion plainly: newer models have improved at long contexts, but the central finding remains. Scores begin falling sharply far below advertised lengths.\n\n\n> [!NOTE]\n> \n> **A Wrinkle When Running the Benchmark Yourself**\n> \n> The [**code**](https://github.com/adobe-research/NoLiMa) and dataset are public. A GitHub user reported that Azure OpenAI's default content filter interrupted some questions after classifying them as violent content. Coauthor Franck Dernoncourt replied that their evaluation had disabled the filter.\n> \n> When reproducing the benchmark, safety-filter blocks may count as incorrect answers too. Check whether the lost points come from the model itself.\n\n\n## Conclusion\n\nNoLiMa removes an easily overlooked shortcut in needle-in-a-haystack tests: the question and needle share no words, only a simple association.\n\nUnder this setup:\n\n1. **Advertised and effective lengths differ greatly.** Of 13 models, only GPT-4o reaches an effective length of 8K. At 32K, 11 fall below half their own base scores.\n2. **The difficulty combines association with length.** Two-hop curves sink across all positions, including the beginning and end. Distance-aligned experiments point to context length as a cause beyond position alone.\n3. **CoT and reasoning models help, but leave a large gap.** All three reasoning models score below 50% at 32K on NoLiMa-Hard.\n4. **Literal clues have a large effect.** Adding choices raises two-hop performance at 32K from 25.9 to 87.2. Adding misleading matching words instead shrinks GPT-4o's effective length to 1K.\n\nThe authors also draw a direct implication for search and RAG: the document containing the answer may use entirely different words from the question, while a document with similar wording but no answer may pull the model off course.\n\nMany models can fit 128K. Far fewer can still find the answer when the question is phrased differently.\n","imageSizes":{"/content/papers/assets/08541b413f077ef8e8f464be627b19590979a35b8c741a89d532e688f2b22b21.png":{"width":1440,"height":919}}}