{"notes":[{"id":"text-spotting/olmocr","category":"text-spotting","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"olmOCR","subtitle":"Slipping the Model a Cheat Sheet","author":"Z. Yuan","paperDate":"2025-02-25","publishedOn":"2026-02-03","paperTitle":"olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models","paperUrl":"https://arxiv.org/abs/2502.18443","bibliography":{"title":"olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models","authors":["Jake Poznanski","Aman Rangapur","Jon Borchardt","Jason Dunkelberger","Regan Huff","Daniel Lin","Aman Rangapur","Christopher Wilhelm","Kyle Lo","Luca Soldaini"],"url":"https://arxiv.org/abs/2502.18443","pdfUrl":"https://arxiv.org/pdf/2502.18443v3","codeUrl":"https://github.com/allenai/olmocr","preprint":{"id":"2502.18443","submittedOn":"2025-02-25","url":"https://arxiv.org/abs/2502.18443"},"publication":null,"citation":"@misc{textspottingolmocr2025,\n  title = {{olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models}},\n  author = {Jake Poznanski and Aman Rangapur and Jon Borchardt and Jason Dunkelberger and Regan Huff and Daniel Lin and Aman Rangapur and Christopher Wilhelm and Kyle Lo and Luca Soldaini},\n  year = {2025},\n  eprint = {2502.18443},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2502.18443}\n}\n","citationPath":"/content/papers/citations/text-spotting/olmocr.48e77a79a2cb.bib"},"thumbnail":{"src":"/content/papers/assets/044c3df431d57e98d00da32fff3b3df21f0cad1cb9a2e9176487efa55b628c17.png","width":994,"height":622},"locale":"en","path":"/content/papers/en/text-spotting/olmocr.74452bf63ce3.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"document-anchoring-give-the-model-the-pdfs-text-too","text":"Document Anchoring: Give the Model the PDF's Text Too","depth":3},{"id":"generating-training-data-with-gpt-4o","text":"Generating Training Data with GPT-4o","depth":3},{"id":"fine-tuning-a-specialized-7b-model","text":"Fine-Tuning a Specialized 7B Model","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"olmocr-bench-scoring-with-unit-tests","text":"olmOCR-Bench: Scoring with Unit Tests","depth":3},{"id":"training-language-models-on-the-output","text":"Training Language Models on the Output","depth":3},{"id":"cost","text":"Cost","depth":3},{"id":"community-reaction","text":"Community Reaction","depth":3},{"id":"what-to-know-when-reading-this-paper","text":"What to Know When Reading This Paper","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"The internet holds trillions of PDFs. Papers, reports, manuals, government documents: much of this content exists only in PDFs and can't be found on any web page. For teams training language models, it is an untapped mine.\n\nThe trouble is that PDFs were never designed for “reading out the text.”\n\nThe format was originally meant to describe “how to print a document on paper.” What the file stores is a series of glyphs, each with its font, size, and coordinates, rather than meaningful units such as headings and paragraphs. Figure 2 of the paper gives an example: the title “olmOCR” is six records in the PDF, one per letter, each carrying its own transformation matrix. Which character to read first, and which paragraph follows which, is not the PDF's concern.\n\nThe Allen Institute for AI (Ai2) introduced olmOCR in February 2025 with the goal of converting large numbers of PDFs into clean plain text in reading order. This note follows v3 from July 2025. The model, training data, and code are all open source under permissive licenses.\n\n## Defining the Problem\n\nApproaches at the time fell roughly into two camps:\n\n- **Pipeline tools** such as MinerU and Marker chain together several models for layout segmentation, table parsing, and so on; tools like GROBID and PaperMage specialize in academic papers.\n- **End-to-end models** such as Nougat and GOT take a page image and output text directly.\n\nThe authors point out that by looking only at the image, end-to-end models give something up: **many PDFs are born digital and already contain the text**. That text may be jumbled, but the characters are correct.\n\nOn the other hand, simply having a general-purpose VLM like GPT-4o read the page wasn't good enough either. The authors observed that on text-dense or complex-layout pages, GPT-4o working from the image alone tends to omit content, rewrite or “complete” the original, and caption images when nobody asked it to. The appendix is more specific: given only the page image, the model finishes incomplete sentences, and when the image is ambiguous, it even invents longer passages.\n\nSo the question becomes: can a VLM look at the image and also receive the PDF's text as a reference?\n\n## Solving the Problem\n\n### Document Anchoring: Give the Model the PDF's Text Too\n\n[![Document anchoring illustration: on the left, a PDF page; on the right, the anchor text extracted from it, including the page dimensions, image positions, and text blocks with coordinates such as “[71x459]Figure 1” and “[71x398]1 Introduction”](/content/papers/assets/044c3df431d57e98d00da32fff3b3df21f0cad1cb9a2e9176487efa55b628c17.png)](/content/papers/assets/044c3df431d57e98d00da32fff3b3df21f0cad1cb9a2e9176487efa55b628c17.png)\n\n*Figure 3 from the paper, excerpted from [**page 17 of the v3 PDF**](https://arxiv.org/pdf/2502.18443v3#page=17), CC BY 4.0. The PDF page is on the left; on the right is the anchor text inserted into the prompt.*\n\nThe box on the right is what the authors call **document anchoring**. It works like this:\n\n1. Use pypdf to read the PDF's internal structure and extract every text block and image, along with their coordinates on the page.\n2. Select blocks for the prompt in priority order until a character limit is reached. Blocks at the beginning and end of the document come first, since titles, abstracts, and footer information usually sit there.\n3. Give this anchor text to the VLM together with the page image, and ask it to output the page's plain text.\n\nLook at the format in the figure: `[71x459]Figure 1` means there is a text block reading “Figure 1” at coordinates (71, 459); `[Image 480x298 to 525x312]` marks the position of an image.\n\nThe authors say frankly that this anchor text is **very noisy**: it doesn't preserve reading order, and the body text is mixed in with all kinds of boilerplate and noise from PDF rendering. But it gives the model characters it can copy, along with their approximate positions on the page. The VLM's job is to understand the layout and put things in order; the PDF text's job is to pin the characters down.\n\nIf the page is a scan with no embedded text, the anchor text shrinks to the page dimensions and image positions, and the model reads mainly from the image.\n\n### Generating Training Data with GPT-4o\n\nWith the anchoring method in hand, the authors first used it to build training data.\n\nIn October 2024, they compared several candidate “teacher” models: Gemini 1.5 frequently returned RECITATION errors, GPT-4o mini hallucinated too much, and Claude Sonnet 3.5 was too expensive. They settled on `gpt-4o-2024-08-06`.\n\nThe prompt asks GPT-4o to answer in structured JSON: first metadata about the page, such as its language, orientation, and whether it contains tables, then the text in reading order. The authors found this structure crucial; it keeps GPT-4o from making up image captions when a page has no text.\n\nThe resulting dataset, **olmOCR-mix-0225**, has 258,641 pages from 102,825 documents: 96,929 sampled from about 240 million public web PDFs, plus 5,896 public-domain books from the Internet Archive. Based on a sample, academic documents make up about 56%, followed by brochures and legal documents.\n\n### Fine-Tuning a Specialized 7B Model\n\nThe anchored prompt worked well for steering GPT-4o, but the authors found that its effect varies from model to model, which makes it better suited as a data-generation technique. Their next question: can a smaller specialized VLM do as well as GPT-4o?\n\nStarting from Qwen2-VL-7B-Instruct, they fine-tuned on olmOCR-mix-0225 for about 1.2 epochs, training for 16 hours on one machine with eight H100s. During fine-tuning, pages are resized so that the longest side is 1,024 pixels, and the anchor text is capped at 6,000 characters. A typical page input is about 1,000 tokens for the image and about 1,800 for the anchor text, around 3,000 in total.\n\n## Discussion\n\n### olmOCR-Bench: Scoring with Unit Tests\n\nTo compare OCR systems, the authors didn't want metrics like edit distance that measure “how close the output is to the reference.” Their example is math: the strings for $x$ and $x_i$ differ only slightly, yet their meanings are completely different.\n\nSo they built **olmOCR-Bench**: about 1,400 PDFs and 7,010 pass/fail unit tests, each of which a program can decide unambiguously:\n\n- **Text presence**: whether a passage of one to three sentences appears in the output.\n- **Text absence**: whether peripheral content such as headers, footers, and page numbers has been left out.\n- **Reading order**: whether two passages appear in the correct order. On a page with several news stories, for example, a test can check only that each story's headline comes before its first sentence, without caring how the stories are ordered relative to each other.\n- **Tables**: a cell with a given value exists, and the cells above it and to its left are as expected.\n- **Math**: render the reference LaTeX with KaTeX, record the relative position of each symbol, then check whether the output contains a set of symbols arranged the same way.\n- **Baseline**: present on every page; it checks that some text was output, that the ending contains no more than 30 repeated n-grams, and that there are no Chinese, Japanese, or emoji characters.\n\nThe baseline check effectively turns “falling into repetition” and “text that shouldn't be there” into mandatory questions on every exam.\n\n\n> [!NOTE]\n> \n> The paper gives three different sizes for this benchmark: 1,400 PDFs in the abstract, 1,402 in §3, and 1,403 in the conclusion. The number of tests is 7,010 throughout.\n\n\n[![olmOCR-Bench results table: pass rates and overall scores for GOT OCR, Marker, MinerU, Mistral OCR API, GPT-4o, Gemini Flash 2, Qwen 2 VL, Qwen 2.5 VL, and olmOCR across eight sources: ArXiv, old scans math, tables, old scans, headers and footers, multi-column, long tiny text, and baseline](/content/papers/assets/970cd043bba14de362481d37c8428e947780e9ec1c949f7846a8be7c4f58fbbf.png)](/content/papers/assets/970cd043bba14de362481d37c8428e947780e9ec1c949f7846a8be7c4f58fbbf.png)\n\n*Table 4 from the paper, excerpted from [**page 8 of the v3 PDF**](https://arxiv.org/pdf/2502.18443v3#page=8), CC BY 4.0. The columns are, in order, ArXiv, Old Scans Math, Tables, Old Scans, Headers & Footers, Multi-Column, Long Tiny Text, and Baseline; the overall score is the average across sources, with a 95% confidence interval.*\n\nThe highest overall score goes to anchored olmOCR at 75.5; the commercial Mistral OCR API scores 72.0, Marker 70.1, MinerU 61.5, and GOT only 48.3. The authors make a point of noting that olmOCR-Bench was built **after** the model was trained, to avoid tuning the model against the benchmark again and again.\n\nOne comparison in the table deserves a closer look: **anchoring helps different models by very different amounts**.\n\n| Model | Without anchoring | With anchoring | Difference |\n| --- | ---: | ---: | ---: |\n| GPT-4o | 68.9 | 69.9 | +1.0 |\n| Gemini Flash 2 | 57.8 | 63.8 | +6.0 |\n| olmOCR | 74.7 | 75.5 | +0.8 |\n\nFor olmOCR itself, anchoring adds only 0.8 points, roughly the size of the confidence interval. Its real progress comes mainly from the training data built with anchoring: the same Qwen2-VL base model scores only 31.5 without fine-tuning.\n\nAlso, olmOCR scores only 42.2 in the Old Scans column and around 71.0 on tables. Scans have no embedded text to anchor to, and they remain the hard part.\n\n### Training Language Models on the Output\n\nThe authors also tested whether this text is useful for language models. They took an intermediate checkpoint of OLMo-2-7B and continued pretraining on 50 billion tokens in each of two runs: one on the peS2o academic papers originally extracted with GROBID plus rules, the other on the same PDFs reprocessed with olmOCR.\n\nAcross six tasks, including MMLU, ARC, DROP, and HellaSwag, the average rose from 53.9 to 55.2, a gain of 1.3 percentage points.\n\n### Cost\n\n| System | Cost per million pages (USD) |\n| --- | ---: |\n| GPT-4o (batch) | 6,240 |\n| Marker (H100) | 1,484 |\n| Mistral OCR API | 1,000 |\n| MinerU (L40S) | 596 |\n| Gemini Flash 2 (batch) | 249 |\n| olmOCR (L40S) | 176 |\n\n*Table: Excerpted from Table 6 of the paper. Estimates assume 0.79 US dollars per hour for an L40S and 2.69 US dollars per hour for an H100; the olmOCR figure includes a 12% retry rate.*\n\nThe abstract's “a million pages for only 176 US dollars” refers to the estimate for running inference yourself on an L40S.\n\n### Community Reaction\n\nolmOCR reached Hacker News on February 28, 2025, and someone in the thread went straight to generative OCR's old problems: hallucinations and inconsistent results. User vikp, who develops Marker, replied that his team had seen olmOCR drop a fair amount of text and hallucinate on their own test set, and he questioned the paper for comparing Marker's API price with olmOCR's self-hosted inference cost, an unequal comparison. [Hacker News, 2025-02-28](https://news.ycombinator.com/item?id=43174298)\n\nvikp develops a competing product, and his position should be weighed accordingly; still, the comparison conditions he points out are visible in Table 6: Marker is listed in forced-OCR mode on an H100, olmOCR on an L40S.\n\n### What to Know When Reading This Paper\n\n1. **The training data comes from GPT-4o.** olmOCR learns GPT-4o's output under the anchored prompt, so it may pick up the teacher's mistakes as well. The authors separately measure how closely student and teacher agree in the appendix.\n2. **The benchmark is their own.** The design of olmOCR-Bench is a useful reference, but the first comparison results come from the authors themselves.\n3. **Anchoring depends on PDF quality.** Born-digital PDFs work best; pure scans have no embedded text to use.\n\n## Conclusion\n\nolmOCR rests on a very plain idea: **the PDF already has the text, so show it to the model**.\n\n1. **Document anchoring** puts the PDF's embedded text and coordinates into the prompt, giving the VLM characters it can copy.\n2. **Build data with anchoring, then fine-tune a small model**: 260,000 pages labeled by GPT-4o plus a 7B Qwen2-VL produce an overall score higher than GPT-4o's own.\n3. **olmOCR-Bench replaces similarity with unit tests**, turning repetition, missing text, and reading order into questions that can be graded automatically.\n\nThe biggest risk with generative OCR is that it fills in characters on its own when it can't see clearly. One of olmOCR's answers: before guessing, look around for characters you don't have to guess.\n","imageSizes":{"/content/papers/assets/044c3df431d57e98d00da32fff3b3df21f0cad1cb9a2e9176487efa55b628c17.png":{"width":994,"height":622},"/content/papers/assets/970cd043bba14de362481d37c8428e947780e9ec1c949f7846a8be7c4f58fbbf.png":{"width":1394,"height":527}}}