{"notes":[{"id":"model-evaluation/lost-in-the-middle","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Lost in the Middle","subtitle":"The Curse of the U","author":"Z. Yuan","paperDate":"2023-07-06","publishedOn":"2026-06-09","world":"external/lost-in-the-middle","paperTitle":"Lost in the Middle: How Language Models Use Long Contexts","paperUrl":"https://arxiv.org/abs/2307.03172","bibliography":{"title":"Lost in the Middle: How Language Models Use Long Contexts","authors":["Nelson F. Liu","Kevin Lin","John Hewitt","Ashwin Paranjape","Michele Bevilacqua","Fabio Petroni","Percy Liang"],"url":"https://arxiv.org/abs/2307.03172","pdfUrl":"https://arxiv.org/pdf/2307.03172v3","codeUrl":null,"preprint":{"id":"2307.03172","submittedOn":"2023-07-06","url":"https://arxiv.org/abs/2307.03172"},"publication":{"venue":"Transactions of the Association for Computational Linguistics","year":2024,"kind":"journal","url":"https://aclanthology.org/2024.tacl-1.9/"},"citation":"@misc{modelevaluationlostinthemiddle2023,\n  title = {{Lost in the Middle: How Language Models Use Long Contexts}},\n  author = {Nelson F. Liu and Kevin Lin and John Hewitt and Ashwin Paranjape and Michele Bevilacqua and Fabio Petroni and Percy Liang},\n  year = {2023},\n  eprint = {2307.03172},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2307.03172}\n}\n","citationPath":"/content/papers/citations/model-evaluation/lost-in-the-middle.77681ac150c1.bib"},"thumbnail":{"src":"/content/papers/assets/14c392ed0cd20cff36405d2cccc1e1d2e317e1c84de40c4eae37cde79f4a31e3.png","width":1278,"height":425},"locale":"en","path":"/content/papers/en/model-evaluation/lost-in-the-middle.27abe25573b3.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"multi-document-question-answering","text":"Multi-Document Question Answering","depth":3},{"id":"keyvalue-retrieval","text":"Key–Value Retrieval","depth":3},{"id":"models-tested","text":"Models Tested","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"the-u-shaped-curve","text":"The U-Shaped Curve","depth":3},{"id":"are-extended-models-better-at-using-long-contexts","text":"Are Extended Models Better at Using Long Contexts?","depth":3},{"id":"even-copying-a-string-can-go-wrong","text":"Even Copying a String Can Go Wrong","depth":3},{"id":"why-does-this-happen","text":"Why Does This Happen?","depth":3},{"id":"what-about-adding-more-documents","text":"What About Adding More Documents?","depth":3},{"id":"what-happened-next","text":"What Happened Next?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-evaluation/nolima","relation":"citing","topic":""}]},{"id":"model-evaluation/nolima","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"NoLiMa","subtitle":"Reading Between the Lines","author":"Z. Yuan","paperDate":"2025-02-07","publishedOn":"2026-07-30","paperTitle":"NoLiMa: Long-Context Evaluation Beyond Literal Matching","paperUrl":"https://arxiv.org/abs/2502.05167","bibliography":{"title":"NoLiMa: Long-Context Evaluation Beyond Literal Matching","authors":["Ali Modarressi","Hanieh Deilamsalehy","Franck Dernoncourt","Trung Bui","Ryan Rossi","Seunghyun Yoon","Hinrich Schütze"],"url":"https://arxiv.org/abs/2502.05167","pdfUrl":"https://arxiv.org/pdf/2502.05167v3","codeUrl":"https://github.com/adobe-research/NoLiMa","preprint":{"id":"2502.05167","submittedOn":"2025-02-07","url":"https://arxiv.org/abs/2502.05167"},"publication":{"venue":"ICML","year":2025,"kind":"conference","url":"https://proceedings.mlr.press/v267/modarressi25a.html"},"citation":"@misc{modelevaluationnolima2025,\n  title = {{NoLiMa: Long-Context Evaluation Beyond Literal Matching}},\n  author = {Ali Modarressi and Hanieh Deilamsalehy and Franck Dernoncourt and Trung Bui and Ryan Rossi and Seunghyun Yoon and Hinrich Schütze},\n  year = {2025},\n  eprint = {2502.05167},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.05167}\n}\n","citationPath":"/content/papers/citations/model-evaluation/nolima.264a4512e69f.bib"},"thumbnail":{"src":"/content/papers/assets/d3917196027c3729a255d140f88c8ddc4cf99d5ea087de73ead2f387a8befb47.png","width":1440,"height":919},"locale":"en","path":"/content/papers/en/model-evaluation/nolima.8d1212b266fc.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"a-needle-that-cannot-be-found-by-matching-words","text":"A Needle That Cannot Be Found by Matching Words","depth":3},{"id":"cleaning-the-hay","text":"Cleaning the Hay","depth":3},{"id":"how-is-it-scored","text":"How Is It Scored?","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"main-results","text":"Main Results","depth":3},{"id":"two-hops-are-harder-than-one-inverted-order-is-harder-than-default","text":"Two Hops Are Harder Than One; Inverted Order Is Harder Than Default","depth":3},{"id":"is-the-problem-position-or-length","text":"Is the Problem Position or Length?","depth":3},{"id":"does-thinking-help","text":"Does Thinking Help?","depth":3},{"id":"put-the-literal-clues-back","text":"Put the Literal Clues Back","depth":3},{"id":"what-about-newer-models","text":"What About Newer Models?","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"model-evaluation/lost-in-the-middle","relation":"referenced","topic":""}]}],"text":"In 2023, language models' context lengths kept growing.\n\nGPT-3.5-Turbo gained a 16K version, Claude gained a 100K version, and the open-source community was busy extending LLaMA's 2,048 tokens to 16K. The advertised numbers kept getting bigger, and a natural thought followed: if more documents fit, why not put them in?\n\nBut fitting something in and making use of it are two different things.\n\nDoes a model have the same chance of using a document placed first in a prompt as one placed tenth?\n\nThis paper from Stanford, UC Berkeley, and Samaya AI, later published in TACL, measures exactly that. The resulting curve is U-shaped: models perform best when the answer is at the beginning or end, and noticeably worse when it is in the middle.\n\n## Defining the Problem\n\nThe authors ask a straightforward question:\n\n- **Can language models reliably use information anywhere in a long context?**\n\nIf they can, moving relevant information from the first passage to the tenth should barely change performance.\n\nThat gives us a clean experimental design. Keep the question and answer unchanged, and vary just two things:\n\n1. **Where the relevant information appears.**\n2. **How long the context is.**\n\nHold everything else fixed and watch the scores.\n\n## Solving the Problem\n\nThis is an evaluation paper. “Solving” the problem here means designing experiments that isolate and measure the position effect. The authors prepare two tasks: one close to a practical RAG scenario, and another that removes as much meaning as possible, leaving only “find the identical string.”\n\n### Multi-Document Question Answering\n\nThe first task imitates generative search as it was commonly implemented at the time: give the model a question and $k$ documents. **Only one** contains the answer; the remaining $k-1$ are distractors.\n\nThe input looks roughly like this, excerpted from Figure 2 of the paper with omissions:\n\n```text\nWrite a high-quality answer for the given question using only the provided search\nresults (some of which might be irrelevant).\n\nDocument [1](Title: Asian Americans in science and technology) Prize in physics for\ndiscovery of the subatomic particle J/ψ. Subrahmanyan Chandrasekhar shared...\nDocument [2](Title: List of Nobel laureates in Physics) The first Nobel Prize in\nPhysics was awarded in 1901 to Wilhelm Conrad Röntgen, of Germany, who received...\nDocument [3](Title: Scientist) and pursued through a unique method, was essentially\nin place. Ramón y Cajal won the Nobel Prize in 1906 for his remarkable...\n\nQuestion: who got the first nobel prize in physics\nAnswer:\n```\n\nThe correct answer, Wilhelm Conrad Röntgen, is hidden in Document [2].\n\nThe questions come from NaturalQuestions-Open: real Google search queries paired with answers manually annotated in Wikipedia. The authors select 2,655 questions whose annotated answers are full paragraphs.\n\nThe distractors are chosen deliberately. The authors use Contriever to retrieve Wikipedia passages that are **highly relevant to the question but do not contain the answer**, then order them by relevance. These passages all look plausible; the model must actually read them to tell them apart.\n\nNow the authors can vary the conditions:\n\n- **Position**: Move the answer-containing document to the first, fifth, tenth… position. The content stays exactly the same; only the order changes.\n- **Length**: Increase the distractors from nine to 19 and 29, giving 10, 20, and 30 documents in total—roughly 2K, 4K, and 6K tokens.\n\nScoring is lenient: an output is correct if it contains any accepted reference answer.\n\n\n> [!TIP]\n> \n> **Two More Reference Lines**\n> \n> To put the scores in perspective, the authors measure two baselines:\n> \n> - **Closed-book**: No documents at all; answer using only the model's own knowledge.\n> - **Oracle**: Give only the one document containing the answer.\n> \n> If providing 20 documents produces a score below closed-book, those documents have actively confused the model.\n\n\n### Key–Value Retrieval\n\nThe second task removes meaning and keeps only the most basic act of finding.\n\nGive the model a JSON object containing $k$ key–value pairs, where every key and value is a randomly generated UUID. Then provide a key and ask for its value:\n\n```text\nExtract the value corresponding to the specified key in the JSON object below.\n\nJSON data:\n{\"2a8d601d-1d69-4e64-9f90-8ad825a74195\": \"bb3ba2a5-7de8-434b-a86e-a88bb9fa7289\",\n \"a54e2eed-e625-4570-9f74-3624e77d6684\": \"d1ff29be-4e2a-4208-a182-0cea716be3d4\",\n \"9f4a92b9-5f69-4725-ba1e-403f08dea695\": \"703a7ce5-f17f-4e6d-b895-5836ba5ec71c\",\n ...}\n\nKey: \"9f4a92b9-5f69-4725-ba1e-403f08dea695\"\nCorresponding value:\n```\n\nThe task requires finding an identical string in the context and copying the string after it. No semantic understanding is needed. The authors use 75, 140, and 300 pairs—roughly 4K, 8K, and 16K tokens—with 500 examples per setting.\n\nIf a model struggles even here, the problem is more basic than understanding a document: it cannot find the information.\n\n### Models Tested\n\nThe authors test six models, including two pairs of shorter and extended versions of the same model:\n\n| Model | Maximum context | Notes |\n| --- | --- | --- |\n| GPT-3.5-Turbo | 4K | OpenAI API |\n| GPT-3.5-Turbo (16K) | 16K | Extended version of the same model |\n| Claude-1.3 | 8K | Anthropic API |\n| Claude-1.3 (100K) | 100K | Extended version of the same model |\n| MPT-30B-Instruct | 8K | Open source; uses ALiBi positional encoding |\n| LongChat-13B (16K) | 16K | Open source; extends LLaMA-13B from 2,048 to 16,384 |\n\nThe OpenAI models use the 0613 versions. All models use greedy decoding, choosing the highest-probability token at each step.\n\n## Discussion\n\n### The U-Shaped Curve\n\n[![Accuracy versus the position of the answer document in multi-document QA; most models show a U-shaped curve with 10, 20, and 30 documents](/content/papers/assets/14c392ed0cd20cff36405d2cccc1e1d2e317e1c84de40c4eae37cde79f4a31e3.png)](/content/papers/assets/14c392ed0cd20cff36405d2cccc1e1d2e317e1c84de40c4eae37cde79f4a31e3.png)\n\n*Figure 5 from the paper, excerpted from [**page 5 of the v3 PDF**](https://arxiv.org/pdf/2307.03172v3#page=5). From left to right: 10, 20, and 30 documents. The horizontal axis gives the answer document's position; farther left means closer to the beginning.*\n\nStart with the middle panel: 20 documents, about 4K tokens.\n\nGPT-3.5-Turbo's line starts at the upper left. With the answer in the first document, accuracy is about 76%. Move it rightward and accuracy falls toward a trough near the tenth document, before recovering somewhat at the twentieth.\n\nThe appendix's raw numbers make this clearer:\n\n| Answer document position | 1st | 5th | 10th | 15th | 20th |\n| --- | ---: | ---: | ---: | ---: | ---: |\n| GPT-3.5-Turbo | 75.8% | 57.2% | 53.8% | 55.4% | 63.2% |\n\n*Compiled from Table 6, Appendix G.2, of the paper.*\n\nCompare those numbers with the two reference lines: GPT-3.5-Turbo scores 56.1% closed-book and 88.3% with the oracle document.\n\nWith the answer in the tenth document, 53.8% is below the score obtained with no documents at all.\n\nTwenty documents are provided, one clearly contains the answer, and the model still performs worse than when given nothing.\n\nThat is the “lost in the middle” of the title. The authors call the advantage at the beginning **primacy bias**, and the advantage at the end **recency bias**.\n\n\n> [!NOTE]\n> \n> **Humans Show Something Similar**\n> \n> Psychology has a “serial-position effect”: when people memorize a list and then freely recall it, they most readily remember items at the beginning and end. The authors mention this connection in their related work.\n> \n> They also note that the phenomenon is somewhat surprising in Transformers. Self-attention can directly access tokens at any position; its design imposes no rule that distant tokens must be harder to see.\n\n\n### Are Extended Models Better at Using Long Contexts?\n\nGPT-3.5-Turbo comes in 4K and 16K versions. The 10- and 20-document settings fit into both.\n\nTheir curves almost overlap. With 20 documents and the answer first, they score 75.8% and 75.7%; with the answer tenth, 53.8% and 54.1%. Claude-1.3 and its 100K version show the same pattern.\n\nThe authors conclude that **the longer and shorter versions use content that fits into both windows in much the same way**. The longer version can hold more material, while its use of that shared content remains similar.\n\n### Even Copying a String Can Go Wrong\n\n[![Key–value retrieval accuracy versus target-key position, with 75, 140, and 300 key–value pairs](/content/papers/assets/d5c6cdc14e580fa77e96f56fa8e44ce6e73030510903d8cd53891982597ae037.png)](/content/papers/assets/d5c6cdc14e580fa77e96f56fa8e44ce6e73030510903d8cd53891982597ae037.png)\n\n*Figure 7 from the paper, excerpted from [**page 7 of the v3 PDF**](https://arxiv.org/pdf/2307.03172v3#page=7). From left to right: 75, 140, and 300 key–value pairs. The horizontal axis gives the target key's position.*\n\nThe key–value results fall into two groups.\n\nClaude-1.3 and Claude-1.3 (100K) stay near 100% in almost every setting, with flat curves along the top.\n\nOther models struggle. Look at the 300-pair panel on the right: GPT-3.5-Turbo (16K) is nearly perfect at the beginning and end, but falls to 45.6% around the 200th pair. Even when all it needs to do is find an exact string, it loses its way in the middle.\n\nLongChat-13B (16K) behaves differently with 140 pairs: the first half is the worst. On inspection, the authors find that when the target key appears near the beginning, the model tends to write code to “query” that key instead of copying its value directly.\n\n### Why Does This Happen?\n\nThe authors run several preliminary analyses.\n\n1. **Model architecture.** The open-source models tested are decoder-only: each token sees only preceding content. The authors additionally test two encoder–decoder models, Flan-T5-XXL and Flan-UL2. Within Flan-UL2's training length of 2,048 tokens, the best and worst positions differ by just 1.9%; beyond that length, the U-shaped curve returns. The authors suggest that a bidirectional encoder lets each document “see” later documents, making it easier to judge which matters.\n\n2. **Putting the question first.** A decoder-only model has not seen the question while reading the documents because the question comes last. What if it also appears before the documents? This makes key–value retrieval almost perfect: GPT-3.5-Turbo (16K) reaches 100% on 300 pairs, versus a worst case of 45.6% before the change. Multi-document QA barely improves, however.\n\n3. **Instruction tuning.** Instructions usually place the task description first. Could that teach a preference for the beginning? Comparing MPT-30B before and after tuning shows a U-shaped curve in both. Instruction tuning actually reduces the best-to-worst position gap from almost 10% to about 4%.\n\n4. **Model size.** Appendix experiments on Llama-2 show only recency bias at 7B; the full U shape appears at 13B and 70B.\n\nSeveral other checks rule out alternative explanations. Random Wikipedia distractors raise overall scores but preserve the U shape. Explicitly saying the search results are randomly ordered, and shuffling the distractors, produces the same trend. GPT-4 outperforms every other model on a 500-question subset, yet its curve is still U-shaped.\n\n### What About Adding More Documents?\n\nFinally, the authors return to a practical RAG setup: use Contriever to retrieve $k$ Wikipedia documents and put all of them into the prompt. This time, there may be no answer-containing document, one, or several.\n\n[![As more documents are retrieved, Contriever's recall keeps rising, but model accuracy plateaus early](/content/papers/assets/e49b7762340b5f14544176911003bf9bd877ecc9933dcaed2d30153d1a280c02.png)](/content/papers/assets/e49b7762340b5f14544176911003bf9bd877ecc9933dcaed2d30153d1a280c02.png)\n\n*Figure 11 from the paper, excerpted from [**page 10 of the v3 PDF**](https://arxiv.org/pdf/2307.03172v3#page=10). The orange dashed line is retriever recall; the other lines are model accuracy.*\n\nThe orange line shows retriever recall: the proportion of questions for which at least one of the top $k$ documents contains the answer. From five to 50 documents, it climbs toward 90.\n\nThe model accuracy curves below it, however, are almost flat after about 20 documents. Increasing from 20 to 50 adds only about 1.5% for GPT-3.5-Turbo and 1% for Claude-1.3, while more than doubling the input length and increasing latency and cost.\n\nThe retriever found it. The model did not use it.\n\nThe authors suggest two directions: **rerank** relevant documents toward the beginning, or **retrieve fewer documents** when appropriate.\n\n### What Happened Next?\n\nInterest in long-context evaluation grew rapidly in the months after this paper appeared.\n\nIn November 2023, Greg Kamradt popularized a needle-in-a-haystack test: insert an out-of-place sentence, such as “the best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day,” into a large collection of Paul Graham essays, then ask what is most fun to do in San Francisco.\n\nIts spirit resembles this paper's key–value retrieval task. But Anthropic's [**December 2023 explanation**](https://claude.com/blog/claude-2-1-prompting) notes that some of Claude 2.1's errors came from its training to avoid answering without sufficient document evidence. Faced with an incongruous sentence, it often said the documents contained no clear answer. Prefilling the answer with “Here is the most relevant sentence in the context:” raised the original evaluation's score from 27% to 98%.\n\nLow scores can have causes beyond position. A model's response tendencies and the wording of a prompt can get mixed into the same number. The closed-book, oracle, random-distractor, and random-order settings in this paper are designed to separate position effects from those other factors.\n\n## Conclusion\n\nBy controlling information position and context length, this paper measures a weakness in how language models use long contexts:\n\n1. **Performance drops sharply when relevant information is in the middle.** With 20 documents, GPT-3.5-Turbo falls from 75.8% at the beginning to 53.8% in the middle, below its 56.1% with no documents.\n2. **Longer-context versions use material that fits much like their shorter counterparts.** The 4K and 16K GPT-3.5-Turbo curves almost overlap.\n3. **Even simple string matching is affected.** With 300 key–value pairs, GPT-3.5-Turbo (16K) falls to 45.6% in the middle.\n4. **The benefit of adding documents saturates quickly.** Increasing from 20 to 50 improves accuracy by only 1% to 1.5%.\n\nThese results come from models available in 2023. The authors turn them into an evaluation criterion: a model claimed to use long contexts reliably should show **little sensitivity to where information appears**, with a small gap between its best and worst positions.\n\nFor RAG practitioners, the practical lesson is that where retrieved documents go, and how many you include, both affect the answer.\n\nWhether today's models still get lost in the middle requires another measurement with today's models.\n","imageSizes":{"/content/papers/assets/14c392ed0cd20cff36405d2cccc1e1d2e317e1c84de40c4eae37cde79f4a31e3.png":{"width":1278,"height":425},"/content/papers/assets/d5c6cdc14e580fa77e96f56fa8e44ce6e73030510903d8cd53891982597ae037.png":{"width":1278,"height":425},"/content/papers/assets/e49b7762340b5f14544176911003bf9bd877ecc9933dcaed2d30153d1a280c02.png":{"width":644,"height":500}}}