{"notes":[{"id":"model-evaluation/faithc4","category":"model-evaluation","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"FaithC4","subtitle":"The Overeager Proofreader","author":"Z. Yuan","paperDate":"2026-05-29","publishedOn":"2026-09-16","paperTitle":"Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models","paperUrl":"https://arxiv.org/abs/2607.21617","bibliography":{"title":"Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models","authors":["Gwang Gook Lee","Kenan Emir Ak","Jay Mohta","Yan Xu","Dimitrios Dimitriadis"],"url":"https://arxiv.org/abs/2607.21617","pdfUrl":"https://arxiv.org/pdf/2607.21617v2","codeUrl":"https://github.com/gwanglee/DoVLMsRead","preprint":{"id":"2607.21617","submittedOn":"2026-05-29","url":"https://arxiv.org/abs/2607.21617"},"publication":null,"citation":"@misc{modelevaluationfaithc42026,\n  title = {{Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models}},\n  author = {Gwang Gook Lee and Kenan Emir Ak and Jay Mohta and Yan Xu and Dimitrios Dimitriadis},\n  year = {2026},\n  eprint = {2607.21617},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2607.21617}\n}\n","citationPath":"/content/papers/citations/model-evaluation/faithc4.b3840dfe13e8.bib"},"thumbnail":{"src":"/content/papers/assets/b3abbe439520c55767dbdc0c5b9cf7cf870e35ff129f6977b59b97671099b945.png","width":1338,"height":338},"locale":"en","path":"/content/papers/en/model-evaluation/faithc4.6fb361c5a3a4.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"starting-from-clean-text","text":"Starting from Clean Text","depth":3},{"id":"15-systems-in-three-categories","text":"15 Systems in Three Categories","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"who-proofreads-the-most","text":"Who Proofreads the Most","depth":3},{"id":"classifying-the-errors","text":"Classifying the Errors","depth":3},{"id":"errors-spread","text":"Errors Spread","depth":3},{"id":"where-does-the-rewriting-happen","text":"Where Does the Rewriting Happen?","depth":3},{"id":"does-telling-it-not-to-correct-help","text":"Does Telling It Not to Correct Help?","depth":3},{"id":"what-to-know-when-reading-this-paper","text":"What to Know When Reading This Paper","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[]}],"text":"Show a VLM an image with “prbolem” printed on it, and it will often answer “problem.”\n\nIf what you want is to understand the content, that's helpful. If what you want is a **verbatim copy**, it's wrong. For legal documents, medical records, and historical manuscripts, when the original contains an error, the transcript should copy the error.\n\nIt's a bit like our own experience with scrambled words: the letters are out of order, yet we can still read them. This paper from Amazon sets out to measure how often VLMs “proofread” the original for you, and at which layer that happens.\n\nThe paper was first uploaded to arXiv on May 29, 2026; this note follows v2 from September 2. The dataset is called **FaithC4**.\n\n## Defining the Problem\n\nCurrent OCR benchmarks such as DocVQA, OCRBench, and OmniDocBench all use existing document images, whose content is mostly correct text. They can measure whether a model reads accurately, but they can't reveal whether a model that meets imperfect text transcribes it faithfully or quietly corrects it.\n\nThe authors also draw a line between two closely related kinds of work:\n\n- KIE-HVQA (2025) deals with **blurred or occluded text**: the visual evidence is insufficient, and the test is whether the model can admit it can't see clearly.\n- FaithC4 deals with **clear but altered text**: the visual evidence is sufficient, and the test is whether the model lets its own expectations override what it sees.\n\n## Solving the Problem\n\n### Starting from Clean Text\n\n[![FaithC4 construction pipeline: text is taken from C4, and words are scrambled, replaced with look-alike characters, or randomly replaced to produce the altered ground truth, which is rendered as a document image and given to a VLM to transcribe](/content/papers/assets/b3abbe439520c55767dbdc0c5b9cf7cf870e35ff129f6977b59b97671099b945.png)](/content/papers/assets/b3abbe439520c55767dbdc0c5b9cf7cf870e35ff129f6977b59b97671099b945.png)\n\n*Figure 1 from the paper, excerpted from [**page 4 of the v2 PDF**](https://arxiv.org/pdf/2607.21617v2#page=4), CC BY 4.0. The example word victory is scrambled into virtcoy, replaced with look-alikes as vjctery, and randomly replaced as xtndovk.*\n\nExisting document images don't let you control the content, so the authors work the other way around: take text first, alter it, then render it.\n\n1. Take passages that fit on one page from the multilingual C4 web corpus: 500 in English, 500 in Chinese, and 455 in Korean, 1,455 in total. English documents average 599 words.\n2. Perturb words of at least 4 characters; **the altered text becomes the ground truth**.\n3. Render it as a single-page PDF image in a 9pt serif font with 1.2 line spacing.\n\nThere are three perturbations, plus the original text as a control:\n\n- **Scramble**: keep the first and last letters and shuffle the ones in between. victory → virtcoy.\n- **Visual**: replace characters with look-alikes. standard → stsndord.\n- **Random**: replace each character with a random letter from the same writing system. standard → xkpmewqi.\n\nChinese has no spaces between words, so it is perturbed in non-overlapping four-character windows, with only the visual and random variants; look-alike characters are limited to those that share a radical or strokes, and random characters are drawn from a list of common characters.\n\nEvery language and condition is calibrated so that about **5% of the characters are changed** (in practice, between 4.65% and 5.20%). This matters: Korean words are short, so if you fixed “how many words to change,” far fewer Korean characters would be altered, Korean would look more robust, and the comparison would be unfair.\n\n### 15 Systems in Three Categories\n\n- **General-purpose VLMs**: Qwen3.5-4B, Qwen3-VL-4B, InternVL3.5-4B, Gemma4-E4B/E2B, Gemini 3 Flash, Gemini 2.5 Flash, GPT-5.4-mini, and others.\n- **OCR-specialized VLMs**: olmOCR-2-7B, DeepSeek-OCR-2, MinerU2.5-Pro, LightOnOCR-2-1B, PaddleOCR, and others.\n- **Traditional OCR**: Tesseract and docTR, which recognize text character by character.\n\nThe prompt is uniformly “extract all the text from the document,” with no warning about typos, so the models answer according to their own habits. OCR-specialized models use their own recommended prompts. The main metric is word error rate (WER), with character error rate used for Chinese. To keep “a model failed on a few documents” out of the comparison, only documents that a model processed successfully under all four conditions are counted.\n\n## Discussion\n\n### Who Proofreads the Most\n\nFor English, here is how much the three perturbations increase WER:\n\n- **Traditional OCR**: at most 0.72 percentage points.\n- **OCR-specialized VLMs**: a wide range, with MinerU at 0.10–0.84, LightOnOCR 0.21–1.40, DeepSeek 0.53–2.47, PaddleOCR 0.64–3.19, and olmOCR 0.76–3.42.\n- **General-purpose VLMs**: mostly 0.59–3.44, with InternVL3.5-4B reaching 6.90. The exception is Gemini 3 Flash, at most 0.23, about the same as traditional OCR.\n\nThe authors' reading: this ranking roughly reflects the strength of each model's language prior. General-purpose VLMs carry language knowledge from broad pretraining; OCR-specialized models have been fine-tuned on documents and have somewhat weaker priors; traditional OCR recognizes character by character, with almost no language model to “correct” it.\n\nWER, though, only tells us how much went wrong, not how.\n\n### Classifying the Errors\n\nThe authors sort the words misread under the scramble condition into four categories:\n\n- **Rewritten**: the output is exactly the original, unaltered word; the most direct case of “the prior overriding the image.”\n- **Close**: within two edits of the altered word, the ordinary noise of any OCR system.\n- **Moderate**: at least half the characters match.\n- **Wrong**: resembles neither the original nor the altered word.\n\n[![FaithC4 Table 5: for each system, the share of misread words under the English scramble condition that fall into the rewritten, close, moderate, and wrong categories, sorted by rewrite rate, from 0% for Tesseract to 58.63% for olmOCR](/content/papers/assets/6ee5629d6762f01ca90d6377a05ca400e408a7d765c225326312b95da772db14.png)](/content/papers/assets/6ee5629d6762f01ca90d6377a05ca400e408a7d765c225326312b95da772db14.png)\n\n*Table 5 from the paper, excerpted from [**page 6 of the v2 PDF**](https://arxiv.org/pdf/2607.21617v2#page=6), CC BY 4.0. Each row sums to 100%; the denominator is the perturbed words that system misread.*\n\nThis table is far more interesting than WER.\n\nWhen traditional OCR misreads, it never changes the word back to the original; most of its errors are “close,” meaning it misread a letter or two. The same goes for Gemini 3 Flash.\n\nThe other models are another story. Of the words InternVL misreads, 46% are changed back to the original; the OCR-specialized LightOn and olmOCR are even higher, at 54% and 59%.\n\nIn other words, **OCR-specialized models make fewer errors overall, but when they do err, there's a good chance they are “proofreading.”** The categories give only a rough picture; the differences between individual models are larger than those between categories.\n\n### Errors Spread\n\nChanging only 5% of the characters affects more than that 5%.\n\nThe authors split the error rate into two parts, “perturbed words” and “unperturbed words.” For Qwen3-VL, the per-word error rate on the original text is 0.40%; under the scramble condition, the error rate on words that were **not perturbed** rises to 2.93%, about 7.3 times as high. For Gemini 3 Flash, it goes from 0.35% to 0.41%, about 1.2 times.\n\nThe abstract says general-purpose VLMs amplify errors by up to 7.3 times; in the same table, however, olmOCR goes from 0.42% to 3.76%, which works out to nearly 9 times.\n\nThat leads to two practical conclusions:\n\n1. **You can't estimate the cost from the perturbation rate alone.** A small share of typos can drag down a much larger part of the output.\n2. **Scores on clean text can't predict this.** Gemini 3 Flash and Qwen3.5-4B have the same 0.35% per-word error rate on the original text, yet after perturbation, how far their errors spread differs severalfold.\n\n### Where Does the Rewriting Happen?\n\nFinally, the authors open up Qwen3-VL-4B (36 layers) to look inside.\n\nFrom each of the 500 English documents, they pick one word at random, highlight it in yellow on the image, and ask the model to “read out the highlighted word,” giving 2,000 cases across the four conditions. For each case, they record two things: the cosine similarity of each layer's FFN output between the original and perturbed versions, and how much of the model's attention falls on that word's image patches when generation begins.\n\nThere are three findings:\n\n1. **The model sees it.** The representations in the first 12 layers are nearly identical; from layer 16 they gradually diverge, and the final-layer similarities are 0.321 for random, 0.469 for visual, and 0.602 for scramble, matching how much each perturbation damages the word's shape. The model also pays more attention to perturbed words than to the originals, reaching 2.14 times as much under scramble at layer 4. As the authors put it, the model notices the perturbation; it just doesn't write what it sees.\n2. **Fully random words don't get rewritten.** In the word probe, 4.9% of scrambled words and 6.0% of visual substitutions are changed back to the original, versus 0% for random replacements: xkpmewqi looks like no real word, so the language prior has no target to change it back to.\n3. **Rewriting happens only when the representation barely changes.** When the scrambled samples are split into four groups by final-layer similarity, the two lowest-similarity groups are never rewritten, while the highest group has a 4.66% rewrite rate. The amount of attention, by contrast, has no bearing on rewriting.\n\nThe authors also find that short words are more likely to be rewritten: up to about 10% for words of 4 to 6 letters, falling sharply at 8 letters or more. Even scrambled, a short word still looks a lot like the original.\n\n### Does Telling It Not to Correct Help?\n\nAppendix F tries the most obvious fix: stating in the prompt that “this document may contain typos, scrambled words, or random strings; transcribe it exactly as written and do not correct anything.”\n\nFor Qwen3-VL-4B and Qwen3.5-4B, this instruction cuts the WER increase under each perturbation by roughly 30% to 50%; for Qwen3-VL under scramble, for example, it falls from 2.04 to 1.07 percentage points. The cost is that the models become more conservative, and WER on the original text actually rises: for Qwen3-VL, from 0.29% to 0.66%.\n\nTurning on thinking mode as well adds almost nothing. Neither model actually produces any substantive reasoning: Qwen3-VL outputs no thinking tokens, and Qwen3.5 gives only empty thinking blocks. The authors conclude that these models treat OCR as a direct perception task, one that doesn't improve by “thinking a little longer.”\n\n### What to Know When Reading This Paper\n\nThe authors list several points in their limitations section:\n\n1. **Only three languages.** Arabic, Devanagari, or domains such as math and code may behave differently.\n2. **The perturbations are artificially controlled**, unlike real degradation such as scanning noise, variation in handwriting, and low resolution.\n3. **The documents are synthetically rendered** and much cleaner than real scans.\n4. **The internal probing covers only one model, Qwen3-VL-4B**; whether other architectures behave the same way remains to be verified.\n5. **The only mitigations tried were prompting and thinking mode**; fine-tuning, decoding-time interventions, and dedicated faithfulness training weren't tested.\n\n## Conclusion\n\nFaithC4 turns “will a VLM change your text for you?” into a measurable question:\n\n1. **Rewriting is widespread, but its degree varies by model.** The English WER increase runs roughly general-purpose VLMs > OCR-specialized VLMs > traditional OCR; Gemini 3 Flash is the exception among general-purpose VLMs.\n2. **They go wrong in different ways.** When traditional OCR errs, it misreads letters; when many VLMs err, they change the word back to one they recognize, which accounts for 59% of olmOCR's errors.\n3. **Errors spread.** Perturbing only 5% of the characters can raise the error rate on unperturbed words severalfold.\n4. **The model sees it but doesn't write it.** The internal representations show that the perturbation is detected; rewriting happens only when the representation stays very close to that of the original word.\n\nFor engineers, the most practical takeaway is the method: take clean text, change a few words yourself, render it as an image, and you can find out whether your model is a reader or a proofreader.\n","imageSizes":{"/content/papers/assets/b3abbe439520c55767dbdc0c5b9cf7cf870e35ff129f6977b59b97671099b945.png":{"width":1338,"height":338},"/content/papers/assets/6ee5629d6762f01ca90d6377a05ca400e408a7d765c225326312b95da772db14.png":{"width":666,"height":486}}}