{"notes":[{"id":"text-recognition/vocabulary-reliance","category":"text-recognition","legacyPath":"","legacySlug":"","legacyPaths":[],"title":"Vocabulary Reliance","subtitle":"Side Effects of Cramming Vocabulary","author":"Z. Yuan","paperDate":"2020-05-08","publishedOn":"2026-03-26","paperTitle":"On Vocabulary Reliance in Scene Text Recognition","paperUrl":"https://arxiv.org/abs/2005.03959","bibliography":{"title":"On Vocabulary Reliance in Scene Text Recognition","authors":["Zhaoyi Wan","Jielei Zhang","Liang Zhang","Jiebo Luo","Cong Yao"],"url":"https://arxiv.org/abs/2005.03959","pdfUrl":"https://arxiv.org/pdf/2005.03959v1","codeUrl":null,"preprint":{"id":"2005.03959","submittedOn":"2020-05-08","url":"https://arxiv.org/abs/2005.03959"},"publication":{"venue":"CVPR","year":2020,"kind":"conference","url":"https://openaccess.thecvf.com/content_CVPR_2020/html/Wan_On_Vocabulary_Reliance_in_Scene_Text_Recognition_CVPR_2020_paper.html"},"citation":"@misc{textrecognitionvocabularyreliance2020,\n  title = {{On Vocabulary Reliance in Scene Text Recognition}},\n  author = {Zhaoyi Wan and Jielei Zhang and Liang Zhang and Jiebo Luo and Cong Yao},\n  year = {2020},\n  eprint = {2005.03959},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2005.03959}\n}\n","citationPath":"/content/papers/citations/text-recognition/vocabulary-reliance.5acdb6f80bed.bib"},"thumbnail":{"src":"/content/papers/assets/566d359ce201c95044d8eb677291acf879055af5097aafccc62b096bb2e720ff.png","width":633,"height":233},"locale":"en","path":"/content/papers/en/text-recognition/vocabulary-reliance.5b1badebd408.md","headings":[{"id":"defining-the-problem","text":"Defining the Problem","depth":2},{"id":"solving-the-problem","text":"Solving the Problem","depth":2},{"id":"breaking-it-into-four-stages","text":"Breaking It into Four Stages","depth":3},{"id":"three-training-sets-with-a-controlled-vocabulary","text":"Three Training Sets with a Controlled Vocabulary","depth":3},{"id":"four-metrics","text":"Four Metrics","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"mix-in-random-strings-and-reliance-drops","text":"Mix In Random Strings, and Reliance Drops","depth":3},{"id":"three-prediction-layers","text":"Three Prediction Layers","depth":3},{"id":"context-modules-are-tied-to-the-prediction-layer","text":"Context Modules Are Tied to the Prediction Layer","depth":3},{"id":"letting-two-models-learn-from-each-other","text":"Letting Two Models Learn from Each Other","depth":3},{"id":"what-to-know-when-reading-this-paper","text":"What to Know When Reading This Paper","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"text-recognition/wwwstr","relation":"referenced","topic":""}]},{"id":"text-recognition/wwwstr","category":"text-recognition","legacyPath":"text-recognition/1904-wwwstr","legacySlug":"1904-wwwstr","legacyPaths":[],"title":"WWWSTR","subtitle":"Data and Model Analysis","author":"Z. Yuan","paperDate":"2019-04-03","publishedOn":"2024-10-10","paperTitle":"What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis","paperUrl":"https://arxiv.org/abs/1904.01906","bibliography":{"title":"What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis","authors":["Jeonghun Baek","Geewook Kim","Junyeop Lee","Sungrae Park","Dongyoon Han","Sangdoo Yun","Seong Joon Oh","Hwalsuk Lee"],"url":"https://arxiv.org/abs/1904.01906","pdfUrl":"https://arxiv.org/pdf/1904.01906","codeUrl":null,"preprint":{"id":"1904.01906","submittedOn":"2019-04-03","url":"https://arxiv.org/abs/1904.01906"},"publication":{"venue":"ICCV","year":2019,"kind":"conference","url":"https://arxiv.org/abs/1904.01906"},"citation":"@misc{textrecognitionwwwstr2019,\n  title = {{What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis}},\n  author = {Jeonghun Baek and Geewook Kim and Junyeop Lee and Sungrae Park and Dongyoon Han and Sangdoo Yun and Seong Joon Oh and Hwalsuk Lee},\n  year = {2019},\n  eprint = {1904.01906},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/1904.01906}\n}\n","citationPath":"/content/papers/citations/text-recognition/wwwstr.4537315eda97.bib"},"thumbnail":{"src":"/content/papers/assets/3c14afb4e2ad3bd5c433008345218e2aa4c6c6157b4ef4c546eb632039e4c6a7.jpg","width":1980,"height":220},"locale":"en","path":"/content/papers/en/text-recognition/wwwstr.028814c0629b.md","headings":[{"id":"dataset-analysis","text":"Dataset Analysis","depth":2},{"id":"training-datasets","text":"Training Datasets","depth":3},{"id":"testing-datasets","text":"Testing Datasets","depth":3},{"id":"version-differences","text":"Version Differences","depth":3},{"id":"architecture-analysis","text":"Architecture Analysis","depth":2},{"id":"transformation-stage","text":"Transformation Stage","depth":3},{"id":"feature-extraction-stage","text":"Feature Extraction Stage","depth":3},{"id":"sequence-modeling-stage","text":"Sequence Modeling Stage","depth":3},{"id":"prediction-stage","text":"Prediction Stage","depth":3},{"id":"experimental-setup","text":"Experimental Setup","depth":2},{"id":"discussion","text":"Discussion","depth":2},{"id":"accuracy-time-trade-off-analysis","text":"Accuracy-Time Trade-off Analysis","depth":3},{"id":"accuracy-memory-trade-off-analysis","text":"Accuracy-Memory Trade-off Analysis","depth":3},{"id":"module-analysis","text":"Module Analysis","depth":3},{"id":"failure-case-analysis","text":"Failure Case Analysis","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"text-recognition/yatr","relation":"next","topic":"Training data and generalization in text recognition"}]}],"text":"Scene text recognition scores climbed fast in the second half of the 2010s. The paper opens with an example: on IIIT-5k without a lexicon, accuracy rose from 78.2% to 96.0% in a very short time.\n\nWith scores that high, are the models really seeing the words clearly?\n\nThis paper, from Megvii, China University of Mining and Technology, and the University of Rochester, finds that the strongest models read “words seen during training” especially well and “unseen words” noticeably worse, and that the gap is unrelated to image quality. The authors call this phenomenon **vocabulary reliance**.\n\nThe paper appeared at CVPR 2020; this note follows arXiv v1.\n\n## Defining the Problem\n\n[![Four scene text images with recognition results: on the left, the blurry BEAUTY and everyday are read correctly; on the right, the clear UNIVERSITI is read as UNIVERSITY and QINGYUANLU as AINGYUANLU, with the wrong letters marked in red](/content/papers/assets/566d359ce201c95044d8eb677291acf879055af5097aafccc62b096bb2e720ff.png)](/content/papers/assets/566d359ce201c95044d8eb677291acf879055af5097aafccc62b096bb2e720ff.png)\n\n*Figure 1 from the paper, excerpted from [**page 1 of the v1 PDF**](https://arxiv.org/pdf/2005.03959v1#page=1). In-vocabulary words are on the left, out-of-vocabulary words on the right; red letters are recognition errors.*\n\nStart with these four images. The two on the left are very blurry, and the right half of BEAUTY is barely visible, yet the model reads both correctly. The two on the right are much clearer, and the model gets them wrong: UNIVERSITI on a sign becomes UNIVERSITY, and QINGYUANLU on a street sign becomes AINGYUANLU.\n\nThe recognizer in the figure is ASTER, one of the best-performing models of its time. The authors describe its recurrent memory mechanism as a double-edged sword: words it has seen can be read back even from damaged images, while words it has never seen may be misread even when the image is clear.\n\nAre these isolated cases or a general pattern? Using the same ResNet-50 backbone and SynthText training data, the authors reimplemented four representative methods, then split the IIIT-5k test images into two groups by whether the word appears in the vocabulary:\n\n| Method | All | In-vocabulary | Out-of-vocabulary |\n| --- | ---: | ---: | ---: |\n| CRNN | 86.8 | 91.1 | 68.7 |\n| FAN | 89.9 | 93.1 | 75.3 |\n| CA-FCN | 89.3 | 91.6 | 76.3 |\n| ASTER | 89.2 | 92.9 | 74.6 |\n\n*Table 1 from the paper: IIIT-5k accuracy (%).*\n\nAll four methods lose roughly 15 to 22 percentage points on out-of-vocabulary words. The authors' conclusion: **vocabulary reliance is ubiquitous; only its degree varies**.\n\n## Solving the Problem\n\nTo compare “degrees,” you first need an experimental framework that can separate the variables.\n\n### Breaking It into Four Stages\n\nThe authors follow the decomposition from [What Is Wrong With Scene Text Recognition Model Comparisons?](https://arxiv.org/abs/1904.01906) and split a recognition model into four stages: **transformation** (TRAN), **feature extraction** (FEAT), **context modeling** (CNTX), and **prediction** (PRED).\n\nThe last two stages matter most for vocabulary reliance, so the first two are held fixed: no transformation, and ResNet-50 as the backbone throughout. For the last two, three prediction layers and three context modules can be combined:\n\n- **Prediction layers**:\n  - **Attention decoder** (Atten.): outputs one character at a time, consulting the characters already output at every step.\n  - **CTC**: each feature column outputs its own character; repeats are then merged and blanks removed.\n  - **Segmentation-based** (Seg.): segments each character and reads the characters and their positions from the segmentation map, as in CA-FCN.\n- **Context modules**:\n  - None.\n  - **Bidirectional LSTM** (BLSTM): sweeps the feature sequence once left to right and once right to left.\n  - **Pyramid pooling** (PPM): pools the feature map at several sizes, upsamples each back to the original size and concatenates them, adding global information over different ranges. Segmentation-based methods are incompatible with BLSTM, so they use PPM.\n\n\n> [!TIP]\n> \n> For this four-stage decomposition and how it makes fair comparisons between models possible, see our note:\n> \n> - **[[19.04] WWWSTR: Data and Model Analysis](/blog/papers/text-recognition/wwwstr)**\n\n\n### Three Training Sets with a Controlled Vocabulary\n\nSynthText draws its vocabulary from a large text corpus of tens of thousands of words, which hides any vocabulary effect. The authors synthesize their own data instead.\n\nThey collect the ground-truth words of five test sets, IC13, IC15, SVT, SVTP, and CUTE, for a total of 3,172 unique words, and treat them as the “vocabulary.” IIIT-5k is left out of the collection and is split by whether each word is in this vocabulary: IIIT-I (1,354 images) and IIIT-O (1,646 images).\n\nThey then use the SynthText engine to generate three training sets of about 7 million images each:\n\n- **LexiconSynth (LS)**: every word is drawn from the 3,172-word vocabulary.\n- **RandomSynth (RS)**: fake words made of random characters, with the same length distribution as LS and letters, digits, and punctuation in a ratio of about 6:3:1.\n- **MixedSynth (MS)**: LS and RS mixed at ratio $r$, with $r=0.5$ by default.\n\nEach word is rendered in all caps, all lowercase, and with an initial capital, and has a 10% chance of added punctuation.\n\n### Four Metrics\n\nWith these three datasets, a model's different abilities can be measured separately:\n\n- **OA** (Observation Ability): accuracy on all test images when trained on RS. With no vocabulary to memorize, the score depends entirely on seeing.\n- **VA** (Vocabulary Learning Ability): accuracy on the five in-vocabulary test sets when trained on LS.\n- **VG** (Vocabulary Generalization): the model's gap between IIIT-I and IIIT-O when trained on LS, minus its gap when trained on RS.\n- **HM**: the harmonic mean of the three, used as an overall score.\n\nVG is a clever design, and it is worth stopping to compute once. The IIIT-I and IIIT-O images may differ in difficulty for reasons that have nothing to do with vocabulary. A model trained on RS has never seen a real word, so its gap between the two groups can stand in for the difference in image difficulty itself:\n\n$$\nVG=1-\\big(\\mathrm{Gap}(LS)-\\mathrm{Gap}(RS)\\big)\n$$\n\nTake the attention model without a context module. In Table 3, when trained on LS it scores 93.7% on IIIT-I and 61.0% on IIIT-O, a gap of 32.7 percentage points; when trained on RS, the gap is 9.8. Plug them in: $1-(0.327-0.098)=0.771$, which is the 77.1 in Table 5.\n\nFor CTC, also without a context module, the two gaps are 18.0 and 10.3: $1-(0.180-0.103)\\approx0.923$. It generalizes much better.\n\n## Discussion\n\n### Mix In Random Strings, and Reliance Drops\n\nThe authors first verify the effect of the data. Mixing RS into LS at a given ratio raises the IIIT-5k accuracy of the attention model without a context module from 77.8% to 84.4%; past a ratio of about 0.5, it stops improving. Figure 4 also shows that once RS is mixed in, the probability that the model “answers with an in-vocabulary word” falls as well.\n\nThe vocabulary can't be removed entirely, though. The authors argue that a model still needs a sufficient share of real words to learn vocabulary knowledge that helps recognition.\n\n### Three Prediction Layers\n\n| No. | Prediction | Context | VA | VG | OA | HM |\n| --- | --- | --- | ---: | ---: | ---: | ---: |\n| ① | Atten. | None | 85.7 | 77.1 | 69.6 | 76.9 |\n| ③ | Atten. | BLSTM | 87.0 | 79.1 | 69.8 | 78.0 |\n| ④ | CTC | None | 77.8 | 92.4 | 65.8 | 77.1 |\n| ⑥ | CTC | BLSTM | 79.9 | 90.2 | 67.6 | 78.1 |\n| ⑦ | Seg. | None | 79.7 | 97.3 | 69.9 | 80.8 |\n\n*Table: Excerpted from Table 5 of the paper. HM can be recomputed from the VA, VG, and OA in the same row as a check.*\n\nThree things can be read from the table:\n\n1. **The attention decoder is the best at memorizing words**, with the highest VA and the lowest VG. At every step it takes the characters it has already output as input, so it picks up the vocabulary naturally.\n2. **The segmentation-based approach generalizes best**, with a VG of 97.3, and its observation ability is about the same as the attention decoder's.\n3. **CTC stands out at neither end.** The authors say its performance is largely covered by the other two, and its observation ability is the weakest.\n\n### Context Modules Are Tied to the Prediction Layer\n\nWhether a context module helps depends on what it is paired with.\n\nThe attention decoder already learns vocabulary on its own, so adding a context module changes little. CTC is a different story: with PPM, VA rises from 77.8 to 84.8, and BLSTM also helps (79.9). At the same time, VG falls from 92.4 to 89.5 and 90.2. The authors' explanation is that CTC decoding itself lacks context modeling; the module in front of it fills that gap and brings vocabulary reliance along with it.\n\nHence the authors' recommendation: there is a trade-off between VA and VG. An attention decoder with BLSTM suits applications whose test vocabulary mostly falls within the training range; when you need to read many words the model has never seen, choose a segmentation-based architecture like CA-FCN.\n\n### Letting Two Models Learn from Each Other\n\nFinally, the authors propose a training method that aims to combine the attention decoder's vocabulary ability with the segmentation model's generalization.\n\nThe approach draws on knowledge distillation and deep mutual learning: two models are trained together from scratch, and on top of each one's own loss, a KL-divergence term pulls their output distributions toward each other. The segmentation model relies more on visual features, so for the attention decoder its predictions act as a kind of extra visual regularization.\n\nThe effect is substantial. The attention decoder's VG rises from 77.1% to 93.6%, and its HM from 76.9 to 82.6. When trained on LS, its gap between in-vocabulary and out-of-vocabulary words is nearly halved, from 32.7 percentage points to 16.9.\n\n### What to Know When Reading This Paper\n\n1. **All training data is synthetic.** LS has a vocabulary of only 3,172 words, far smaller than in real applications. It is there to amplify the vocabulary effect, and its numbers can't be converted directly into error rates for ordinary models.\n2. **The models are from 2020.** Later recognizers mostly use larger pretrained language models, so the paper's group-level conclusions need to be re-verified.\n3. **There is no real random-string test set.** The authors note that accuracy on random strings could itself serve as a metric, but at the time no standard dataset combined real-image complexity with purely random labels, so they left it out.\n4. **The tables contain small inconsistencies.** Table 6 lists VA values for the same models that differ from Table 5 (for example, 83.2 versus 85.7 for ①), and the text's claim that PPM raises CTC's VA by 10.5% doesn't match Table 5's numbers either. Table 5's HM can be recomputed from VA, VG, and OA, so this note goes by Table 5.\n\n## Conclusion\n\nThis paper measured a problem that many people had sensed but nobody had quantified:\n\n1. **Vocabulary reliance is ubiquitous.** All four representative methods lose roughly 15 to 22 percentage points on words they have never seen.\n2. **The prediction layer sets the degree of reliance.** The attention decoder memorizes best and generalizes worst; segmentation generalizes best.\n3. **Context modules bring vocabulary in**, most visibly for CTC.\n4. **Mixing in random strings or mutual learning reduces the reliance**, without sacrificing much accuracy.\n\nThe most useful idea in the paper is that metric, which uses random strings to subtract out image difficulty. Six years later, recognizers have become vision-language models that generate whole pages, and a 2026 study that swapped document content for random letters saw the same thing, only at a much larger scale.\n","imageSizes":{"/content/papers/assets/566d359ce201c95044d8eb677291acf879055af5097aafccc62b096bb2e720ff.png":{"width":633,"height":233}}}