{"notes":[{"id":"model-evaluation/low-cer-not-enough","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Low CER","subtitle":"Right Letters, Wrong Person","author":"Z. Yuan","paperDate":"2026-07-27","publishedOn":"2026-10-20","paperTitle":"When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents","paperUrl":"https://arxiv.org/abs/2607.24077","bibliography":{"title":"When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents","authors":["Marina Gardella","Camilo Mariño","Diego Belzarena","Ignacio Ramírez","Gregory Randall","Jean-Michel Morel"],"url":"https://arxiv.org/abs/2607.24077","pdfUrl":"https://arxiv.org/pdf/2607.24077v3","codeUrl":null,"preprint":{"id":"2607.24077","submittedOn":"2026-07-27","url":"https://arxiv.org/abs/2607.24077"},"publication":null,"citation":"@misc{modelevaluationlowcernotenough2026,\n  title = {{When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents}},\n  author = {Marina Gardella and Camilo Mariño and Diego Belzarena and Ignacio Ramírez and Gregory Randall and Jean-Michel Morel},\n  year = {2026},\n  eprint = {2607.24077},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2607.24077}\n}\n","citationPath":"/content/papers/citations/model-evaluation/low-cer-not-enough.4bc2c3f3a7aa.bib"},"thumbnail":{"src":"/content/papers/assets/69d3b045d2c8e7ad52c47a644f8a12435a94a4fb2733c7d1af66965f5c005a49.png","width":1022,"height":261},"locale":"en","path":"/content/papers/en/model-evaluation/low-cer-not-enough.ec469dd2eeeb.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"the-berrutti-dataset","text":"The Berrutti Dataset","depth":3},{"id":"systems-tested","text":"Systems Tested","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"by-standard-metrics-vlms-win","text":"By Standard Metrics, VLMs Win","depth":3},{"id":"page-by-page-four-failures","text":"Page by Page: Four Failures","depth":3},{"id":"why-standard-metrics-miss-it","text":"Why Standard Metrics Miss It","depth":3},{"id":"the-authors-recommendations","text":"The Authors' Recommendations","depth":3},{"id":"what-to-know-when-reading-this-paper","text":"What to Know When Reading This Paper","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"An archival transcript with a character error rate of just 2%. Sounds good.\n\nBut what if those few wrong characters happen to turn one person's name into someone else's?\n\nFor historians, archivists, and human rights investigators, a transcript is part of the evidence. Getting one name wrong can make a person vanish from the record, or put someone else where they don't belong. In the character error rate, this kind of error barely registers.\n\nThis paper comes from the Faculty of Engineering at Uruguay's Universidad de la República, together with Université Paris-Saclay and the Centre Borelli at ENS Paris-Saclay in France, among others. Using archives from Uruguay's military dictatorship, it takes a close look at exactly where VLM transcriptions go wrong. The paper was uploaded to arXiv on July 27, 2026, and has been accepted at the ADAPDA 2026 workshop; this note follows v3 from September 17.\n\n## Defining the Problem\n\nOCR's standard metrics are **character error rate** (CER) and **word error rate** (WER): the number of characters or words that must be inserted, deleted, or substituted to turn the output into the correct answer, divided by the total length.\n\nBoth metrics treat every error the same. Traditional OCR errors are mostly misread letters, so counting this way works fine.\n\nVLMs are different. They generate fluent, plausible text even when the image doesn't hold enough evidence. The authors list several typical cases: completing text from context, making spelling more “correct,” swapping a word for a different one with a similar meaning, and adding content that isn't in the original. Names and dates are hard to infer from context, so they are the most prone to this kind of error, yet the effect on CER and WER is tiny.\n\n## Solving the Problem\n\n### The Berrutti Dataset\n\nThe Berrutti archive is a microfilm collection of more than 3 million pages, mostly produced during Uruguay's military dictatorship (1973 to 1985). It consists mainly of typewritten reports from government agencies, often with handwritten annotations, and is an important resource for human rights investigations and transitional justice in the country.\n\nThe images were scanned straight from the microfilm as binarized black-and-white images and carry real degradation: heavy noise, overlapping visual elements, and scanning defects.\n\nUruguay's National Human Rights Institution authorized the release of 175 of these images for research. The documents were manually reviewed, and sensitive personal data was excluded during selection. Because of the nature of the archive, the authors evaluated only models that can run locally and sent no images to external services.\n\n### Systems Tested\n\n- **Traditional OCR**: PP-OCRv5, EasyOCR, DocTR, Tesseract.\n- **OCR-specialized VLMs**: Surya-OCR, HunyuanOCR, MinerU2.5-Pro, Chandra-OCR 2, GLM-OCR, Qianfan-OCR, dots.mocr, dots.ocr, PaddleOCR-VL 1.5, DeepSeek-OCR 2.\n- **General-purpose VLMs**: Qwen 3.6 (35B, standard and reasoning modes), Gemma 4 (31B), MiniCPM-V 4.5.\n\nEvery model that accepts a custom prompt got the same instruction, and it is explicit: output plain text in reading order; preserve the original spelling, accents, capitalization, punctuation, abbreviations, and dates; **do not correct, do not expand abbreviations, do not invent missing text**; only when something is truly unreadable, write `[ilegible]` (Spanish for “illegible”).\n\n## Discussion\n\n### By Standard Metrics, VLMs Win\n\n[![Character and word error rates for 18 systems on the Berrutti dataset, each given as mean and median; Qwen 3.6 in reasoning mode has the lowest median CER, 1.72%, while the traditional OCR systems Tesseract and EasyOCR have the highest error rates](/content/papers/assets/ead582db9180192c38b48f732e13de82c3082a8d1b67f47e4b8a9cd3876d75dc.png)](/content/papers/assets/ead582db9180192c38b48f732e13de82c3082a8d1b67f47e4b8a9cd3876d75dc.png)\n\n*Table 1 from the paper, excerpted from [**page 7 of the v3 PDF**](https://arxiv.org/pdf/2607.24077v3#page=7). Bold marks the best result in each column; asterisks indicate reasoning mode.*\n\nStart with the two median columns. Qwen 3.6 in reasoning mode has a median CER of 1.72% and a median WER of 5.01%, both the lowest in the table. The best traditional OCR system, DocTR, has a median CER of 5.74%; Tesseract's is 11.86%.\n\nNow look at the gap between mean and median. Qwen 3.6 in reasoning mode has a mean CER of 11.17%, more than six times its median. The authors' interpretation: these models transcribe most documents very accurately but occasionally fail on a whole document, and a handful of pages pull the mean up.\n\nFrom this table alone, the conclusion is simple: VLMs are clearly better than traditional OCR.\n\n### Page by Page: Four Failures\n\nThe authors then went through the outputs page by page and identified four recurring kinds of failure.\n\n**1. Incomplete transcription.** Sometimes a model transcribes only part of the page, leaving out whole passages at the beginning, at the end, or in the middle; sometimes the transcription breaks off midway but generation doesn't stop, degenerating into output that repeats over and over.\n\n**2. Spelling normalization.** Some of the original's spelling errors are quietly corrected by the models, and inconsistently at that: the same model fixes one typo and leaves another. Traditional DocTR, on the other hand, copies the misspelling as written. The authors also point out that traditional OCR, though it copies misspellings, still misreads other letters, which changes the original too. And although Surya rarely alters spelling, when the image is ambiguous it tends to read words as English rather than Spanish.\n\n**3. Spurious additions.** This is the most dangerous kind.\n\n[![A region made completely unreadable by degradation, originally “y se ▒▒▒ hasta la muerte”; models such as MinerU, Qwen 3.6, Chandra, GLM-OCR, DeepSeek, Hunyuan, and Qianfan fill the gap with different words, such as de la de la, llega, mueren, ha manifestada, sigue, and mata](/content/papers/assets/69d3b045d2c8e7ad52c47a644f8a12435a94a4fb2733c7d1af66965f5c005a49.png)](/content/papers/assets/69d3b045d2c8e7ad52c47a644f8a12435a94a4fb2733c7d1af66965f5c005a49.png)\n\n*Fig. 7 from the paper, excerpted from [**page 11 of the v3 PDF**](https://arxiv.org/pdf/2607.24077v3#page=11). The image region is at the top and each model's output below; red text is content that isn't in the original.*\n\nIn the line at the top, the middle word can't be made out at all; only the surrounding “y se … hasta la muerte” remains. The models' outputs all differ: one fills in llega, another mueren, another sigue, another mata.\n\nThis figure shows two things. First, although the prompt explicitly asks for `[ilegible]` when text can't be read, the models still choose to produce a word. Second, different models produce different words for the same region, so these words clearly come from the language prior and not from the image.\n\nThe authors saw a similar problem in lists of names: even where the writing is clear, a VLM may add a few names that aren't in the original. For applications that extract information or build knowledge graphs, that amounts to conjuring several person records out of thin air.\n\n**4. Semantic substitutions and insertions.** A model replaces a word with another that means something different but still fits the grammar, and it often **preserves the entity type**: one person's name becomes another person's name, one country becomes another country.\n\nThe authors stress that most of these errors occur where the image is clear and readable; they have nothing to do with degradation and arise because the model drifts away from the original as it generates. The replacement words are usually close to the originals in both shape and language, so CER and WER rise only slightly, even though the meaning of the sentence has changed.\n\n### Why Standard Metrics Miss It\n\nCER and WER count only “how many characters were changed,” not which ones. Turning a comma into a period and turning one person's name into another's can cost the same on these metrics.\n\nTraditional OCR errors usually look like errors: letters come out crooked, words don't spell anything, and the reader stops to check the original. VLM errors often look perfectly correct and may even read more smoothly than the original.\n\n### The Authors' Recommendations\n\nThe conclusion proposes several directions:\n\n1. **Targeted evaluation**: annotate key information units such as names, dates, and identifiers, score them separately, and distinguish “minor spelling differences” from “serious semantic errors.”\n2. **Uncertainty estimation**: VLMs generate text from probabilities to begin with, so one could study whether spurious or substituted content corresponds to measurable uncertainty, then use that to filter or flag the output.\n3. **Combining systems**: judging from the examples, traditional OCR and VLMs seem to fail in different places and under different conditions, so combining the outputs of several systems might help. The authors add that this needs more experiments to confirm.\n\n### What to Know When Reading This Paper\n\n1. **The dataset is small.** 175 images from a single archive in a single language, and binarized microfilm at that, which is very different from typical scanned documents.\n2. **The failure-mode analysis is qualitative.** The paper illustrates the four error types with many examples but gives no frequency for any of them.\n3. **Only local models were tested.** Cloud services were excluded for privacy reasons, and the paper leaves open whether its conclusions apply to commercial models such as GPT and Gemini.\n\n## Conclusion\n\nThis paper reminds us that “a better score” and “a more reliable transcription” are two different things:\n\n1. **By CER and WER, VLMs clearly win.** Qwen 3.6 in reasoning mode has a median CER of only 1.72%.\n2. **Page by page, VLMs make errors that traditional OCR doesn't**: missing passages, repetition, corrected spelling, words filled into unreadable spots, and names swapped for other names.\n3. **These errors are cheap on the metrics**, but they can be very costly in archival research.\n\nFor any application that treats OCR output as a record or as evidence, the paper's advice is practical: pick out the fields that really matter and evaluate them separately, and make “unreadable” an answer that is respected.\n","imageSizes":{"/content/papers/assets/ead582db9180192c38b48f732e13de82c3082a8d1b67f47e4b8a9cd3876d75dc.png":{"width":1000,"height":672},"/content/papers/assets/69d3b045d2c8e7ad52c47a644f8a12435a94a4fb2733c7d1af66965f5c005a49.png":{"width":1022,"height":261}}}