{"notes":[{"id":"model-evaluation/omnidocbench","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"OmniDocBench","subtitle":"Nine Document Types, One Exam","author":"Z. Yuan","paperDate":"2024-12-10","publishedOn":"2026-10-03","paperTitle":"OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations","paperUrl":"https://arxiv.org/abs/2412.07626","bibliography":{"title":"OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations","authors":["Linke Ouyang","Yuan Qu","Hongbin Zhou","Jiawei Zhu","Rui Zhang","Qunshu Lin","Bin Wang","Zhiyuan Zhao","Man Jiang","Xiaomeng Zhao","Jin Shi","Fan Wu","Pei Chu","Minghao Liu","Zhenxiang Li","Chao Xu","Bo Zhang","Botian Shi","Zhongying Tu","Conghui He"],"url":"https://arxiv.org/abs/2412.07626","pdfUrl":"https://arxiv.org/pdf/2412.07626v2","codeUrl":"https://github.com/opendatalab/OmniDocBench","preprint":{"id":"2412.07626","submittedOn":"2024-12-10","url":"https://arxiv.org/abs/2412.07626"},"publication":{"venue":"CVPR","year":2025,"kind":"conference","url":"https://openaccess.thecvf.com/content/CVPR2025/html/Ouyang_OmniDocBench_Benchmarking_Diverse_PDF_Document_Parsing_with_Comprehensive_Annotations_CVPR_2025_paper.html"},"citation":"@misc{modelevaluationomnidocbench2024,\n  title = {{OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations}},\n  author = {Linke Ouyang and Yuan Qu and Hongbin Zhou and Jiawei Zhu and Rui Zhang and Qunshu Lin and Bin Wang and Zhiyuan Zhao and Man Jiang and Xiaomeng Zhao and Jin Shi and Fan Wu and Pei Chu and Minghao Liu and Zhenxiang Li and Chao Xu and Bo Zhang and Botian Shi and Zhongying Tu and Conghui He},\n  year = {2024},\n  eprint = {2412.07626},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2412.07626}\n}\n","citationPath":"/content/papers/citations/model-evaluation/omnidocbench.6ba6e922e56d.bib"},"thumbnail":{"src":"/content/papers/assets/e020fa9a38c313a3d8657ba9b84fcabe6d57fe8828c5f51d5312e91fb1e1fc01.png","width":688,"height":422},"locale":"en","path":"/content/papers/en/model-evaluation/omnidocbench.0b1f2204cab1.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"data-981-pages-out-of-two-hundred-thousand-pdfs","text":"Data: 981 Pages Out of Two Hundred Thousand PDFs","depth":3},{"id":"scoring-extract-the-elements-then-match","text":"Scoring: Extract the Elements, Then Match","depth":3},{"id":"what-edit-distance-measures","text":"What Edit Distance Measures","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"pipelines-specialized-vlms-and-general-purpose-vlms","text":"Pipelines, Specialized VLMs, and General-Purpose VLMs","depth":3},{"id":"later-versions","text":"Later Versions","depth":3},{"id":"outside-criticism","text":"Outside Criticism","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"By 2024 there were plenty of document parsing models: pipeline tools that chain together layout detection, OCR, and table and formula recognition; end-to-end models like Nougat and GOT that read a whole page in one pass; and general-purpose VLMs such as GPT-4o and Qwen2-VL that could be used off the shelf.\n\nThe question was: how do you compare them?\n\nNougat was tested only on English arXiv papers; GOT's full-page test set had just 90 pages of Chinese and English combined. Everyone used their own data and their own metrics, so even with the numbers side by side, it was hard to tell who was stronger at what.\n\nOmniDocBench, proposed by Shanghai AI Laboratory and other institutions, sets out to be a benchmark that covers many kinds of documents, with complete annotations and fair scoring. The paper appeared at CVPR 2025; this note follows arXiv v2.\n\n## Defining the Problem\n\nThe authors see three shortcomings in existing benchmarks:\n\n1. **Too few document types.** Most contain only academic papers; common documents such as textbooks, exam papers, financial reports, and newspapers rarely appear.\n2. **Unsuitable metrics.** Whole pages are scored with edit distance or BLEU, but formulas and tables have many equivalent forms in LaTeX and HTML, so output that renders identically from a different string still loses points.\n3. **Only one overall score.** It can't show whether a model is weak on text, weak on formulas, or weak only on a certain kind of document.\n\n## Solving the Problem\n\n### Data: 981 Pages Out of Two Hundred Thousand PDFs\n\nThe authors first collected more than 200,000 PDFs from Common Crawl, Google and Baidu searches, and internal data. They extracted image features with ResNet-50, clustered them into 10 groups with Faiss, and picked 6,000 visually diverse pages. Annotators then labeled attributes such as document type, layout, and language, the categories were rebalanced, and **981 pages** remained, spread across nine document types: books, slides, financial reports, textbooks, exam papers, magazines, academic papers, notes, and newspapers.\n\nThe annotations are very fine-grained:\n\n- **Layout**: boxes for 19 block categories, such as titles, paragraphs, tables, and images; attributes for each box, such as the text language, whether there is a complex background, and whether a table has merged cells; reading order; and the relationships that tie figures and tables to their captions and connect paragraphs across pages.\n- **Content**: text as plain text; formulas (including inline formulas) in LaTeX; tables in both HTML and LaTeX.\n\nIn total there are more than 100,000 annotations, including over 20,000 blocks and over 70,000 inline spans.\n\nAnnotation ran in three stages: pre-annotation with LayoutLMv3, PaddleOCR, UniMERNet, and GPT-4o; character-by-character correction by annotators; and finally, CDM's rendering tools to find formulas and tables that failed to render, which three researchers then reviewed.\n\n### Scoring: Extract the Elements, Then Match\n\n[![OmniDocBench scoring pipeline: after preprocessing, the model's Markdown output has LaTeX tables, HTML tables, Markdown tables, code blocks, text blocks, and inline formulas extracted in turn, which are then matched against each element type in the ground truth using Adjacency Search Match](/content/papers/assets/e020fa9a38c313a3d8657ba9b84fcabe6d57fe8828c5f51d5312e91fb1e1fc01.png)](/content/papers/assets/e020fa9a38c313a3d8657ba9b84fcabe6d57fe8828c5f51d5312e91fb1e1fc01.png)\n\n*Figure 4 from the paper, excerpted from [**page 5 of the v2 PDF**](https://arxiv.org/pdf/2412.07626v2#page=5). Model output is on the left, extraction and matching in the middle, and the ground-truth element categories on the right.*\n\nOutput formats vary widely between models: some write tables in HTML, others in Markdown; some write formulas in LaTeX, others in Unicode. Comparing whole-page strings directly isn't fair, so the authors designed a scoring pipeline:\n\n1. **Preprocessing**: remove images, strip Markdown markers at the start of the document, and normalize overly long runs of repeated characters.\n2. **Extraction**: use regular expressions to extract, in order, LaTeX tables, HTML tables, display formulas, Markdown tables (converted to HTML), and code blocks. What remains is plain text, split into paragraphs at blank lines.\n3. **Inline formulas converted to Unicode**: models output inline formulas so inconsistently that the authors don't score them separately; they convert them to Unicode and score them together with the text.\n4. **Matching**: a model may not split paragraphs the way the ground truth does; one paragraph may be cut in two, or two merged into one. The authors propose **Adjacency Search Match**: compute the normalized edit distance between every pair of paragraphs and match pairs whose similarity exceeds a threshold directly; for the rest, use fuzzy matching to check whether one is a substring of the other, and if so, try merging adjacent paragraphs until the edit distance stops improving.\n5. **Ignoring**: headers, footers, page numbers, page footnotes, and figure and table captions are handled too differently from model to model; they take part in matching but aren't scored.\n\nAfter matching, each type of content gets its own metric:\n\n- **Text**: normalized edit distance; lower is better.\n- **Tables**: everything is converted to HTML and scored with TEDS (which measures the similarity of table structure through tree edit distance); higher is better.\n- **Formulas**: CDM, which renders formulas as images and checks whether the character positions line up; higher is better.\n- **Reading order**: normalized edit distance computed over the sequence of text elements.\n\n### What Edit Distance Measures\n\nIt's worth pausing on text edit distance. Normalized edit distance is “how many characters must be changed to reach the correct answer,” divided by the length.\n\nSuppose a page has 3,000 characters, and the model makes just one mistake: it reads the pickup code `K7O2` as `K702`. The edit distance for the page is $1/3000\\approx0.0003$, which barely registers. But if you look only at that 4-character field, the edit distance is $1/4=0.25$, and the field is now useless.\n\nPage-level metrics are good for comparing overall reading ability; when a few key fields are wrong, they hardly show it. That is the scope of the metric itself, and it's worth knowing when you use it.\n\n## Discussion\n\n### Pipelines, Specialized VLMs, and General-Purpose VLMs\n\nThe authors divide the tested methods into three categories:\n\n- **Pipeline tools**: MinerU, Marker, Mathpix.\n- **Specialized VLMs**: GOT-OCR, Nougat.\n- **General-purpose VLMs**: GPT-4o, Qwen2-VL-72B, InternVL2-76B.\n\nOverall, pipeline tools such as MinerU and Mathpix perform best on text, formulas, and tables, and general-purpose VLMs are competitive too. Almost every method does better on English pages than on Chinese ones.\n\nBroken down by the nine document types, things get more interesting:\n\n[![OmniDocBench text edit distance across nine document types: MinerU, Marker, Mathpix, GOT-OCR, Nougat, GPT-4o, Qwen2-VL-72B, and InternVL2-76B on books, slides, financial reports, textbooks, exam papers, magazines, academic papers, notes, and newspapers](/content/papers/assets/13b22f4d607b4a82595fc4d89f10eeb190631df5176a1842c785cbc306f5ea66.png)](/content/papers/assets/13b22f4d607b4a82595fc4d89f10eeb190631df5176a1842c785cbc306f5ea66.png)\n\n*Table 3 from the paper, excerpted from [**page 6 of the v2 PDF**](https://arxiv.org/pdf/2412.07626v2#page=6). Values are normalized edit distance for text; lower is better.*\n\nLook at the two rightmost categories:\n\n| Model | Notes | Newspapers |\n| --- | ---: | ---: |\n| MinerU | 0.984 | 0.171 |\n| GPT-4o | 0.607 | 0.751 |\n| Qwen2-VL-72B | 0.168 | 0.676 |\n| Nougat | 0.991 | 0.871 |\n\nOn handwritten notes, MinerU can barely read anything, while Qwen2-VL-72B is among the best in the whole table. Newspapers are the exact opposite: MinerU scores 0.171, while the general-purpose VLMs are all above 0.67.\n\nThe authors' explanation: general-purpose VLMs are trained on very broad data, so they cope better with unusual formats and with image degradation such as blur, watermarks, and colored backgrounds; but on extremely text-dense pages like newspapers, limits on input resolution and token length often keep VLMs from reading everything. Pipeline tools segment the layout first and process it block by block, so they hold up better on complex layouts.\n\nOn multi-column and complex layouts, every method's reading order gets noticeably worse.\n\nIn their conclusion, the authors suggest that layout-aware designs and domain fine-tuning are promising directions for VLMs. The following year or so went largely that way: small “layout first, then recognition” VLMs such as PaddleOCR-VL and MinerU2.5 climbed to the top of this benchmark.\n\n### Later Versions\n\nOmniDocBench has been revised several times since; the paper describes v1.0:\n\n- **v1.5** (September 2025): adds 374 pages, raises the image resolution of newspapers and notes from 72 DPI to 200 DPI, balances the number of Chinese and English pages, and increases the share of formulas. Chinese and English are no longer scored separately, and the overall score becomes $\\big((1-\\text{text edit distance})\\times100+\\text{table TEDS}+\\text{formula CDM}\\big)/3$.\n- **v1.6** (April 2026): adds 296 harder samples, such as nested tables, dense formulas, and unusual layouts, for 1,651 pages in total; the matching method now adjusts segmentation granularity on the prediction side only.\n\nOn the v1.6 leaderboard in the official README, the top entries are all specialized models of around one billion parameters, with overall scores close to 97. [OmniDocBench README](https://github.com/opendatalab/OmniDocBench)\n\n### Outside Criticism\n\nAs scores approached the ceiling, some began to question whether the benchmark could still tell the top models apart.\n\nIn February 2026, Jerry Liu of LlamaIndex wrote that OmniDocBench was close to saturation and that its documents skewed simple: 1,355 pages and 9 types, weighted toward academic papers, with little attention to complex financial presentations, legal documents, insurance claims, and forms. He also gave examples from his company's own product: a table correctly split into two, or scientific notation written in HTML instead of LaTeX, is semantically right but loses a lot on TEDS and edit distance. He argued for scoring that rewards semantic correctness over an exact format match. [LlamaIndex, 2026-02-24](https://www.llamaindex.ai/blog/omnidocbench-is-saturated-what-s-next-for-ocr-benchmarks)\n\nLlamaIndex has its own document parsing product, and the article's examples all come from its own output, so its position should be weighed accordingly. Still, the paper itself was aware that “a single ground truth is too strict about format”: the authors put a great deal of effort into element extraction, paragraph merging, and ignore rules precisely to reduce misjudgments caused by format differences.\n\n## Conclusion\n\nOmniDocBench makes document parsing evaluation far more fine-grained than before:\n\n1. **Nine document types, 981 pages, and more than 100,000 annotations**, covering layout, reading order, attributes, and the relationships between elements.\n2. **Extract, match, then give each type its own metric**: edit distance for text, TEDS for tables, and CDM for formulas, to keep format differences from skewing the results as much as possible.\n3. **Differences show up only when broken down**: pipeline tools excel at newspapers and financial reports, general-purpose VLMs at handwritten notes and degraded images, and multi-column layouts are hard for everyone.\n\nIts metrics work at the page level and suit comparisons of overall reading ability. Whether a particular number or a particular name was read correctly is a question for separate tests.\n","imageSizes":{"/content/papers/assets/e020fa9a38c313a3d8657ba9b84fcabe6d57fe8828c5f51d5312e91fb1e1fc01.png":{"width":688,"height":422},"/content/papers/assets/13b22f4d607b4a82595fc4d89f10eeb190631df5176a1842c785cbc306f5ea66.png":{"width":1377,"height":319}}}