{"notes":[{"id":"text-recognition/diffusionstr","category":"text-recognition","legacyPath":"text-recognition/2306-diffusionstr","legacySlug":"2306-diffusionstr","legacyPaths":[],"title":"DiffusionSTR","subtitle":"Text Signals Amidst Noise","author":"Z. Yuan","paperDate":"2023-06-29","publishedOn":"2024-10-13","paperTitle":"DiffusionSTR: Diffusion Model for Scene Text Recognition","paperUrl":"https://arxiv.org/abs/2306.16707","bibliography":{"title":"DiffusionSTR: Diffusion Model for Scene Text Recognition","authors":["Masato Fujitake"],"url":"https://arxiv.org/abs/2306.16707","pdfUrl":"https://arxiv.org/pdf/2306.16707","codeUrl":null,"preprint":{"id":"2306.16707","submittedOn":"2023-06-29","url":"https://arxiv.org/abs/2306.16707"},"publication":{"venue":"ICIP","year":2023,"kind":"conference","url":"https://arxiv.org/abs/2306.16707"},"citation":"@misc{textrecognitiondiffusionstr2023,\n  title = {{DiffusionSTR: Diffusion Model for Scene Text Recognition}},\n  author = {Masato Fujitake},\n  year = {2023},\n  eprint = {2306.16707},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2306.16707}\n}\n","citationPath":"/content/papers/citations/text-recognition/diffusionstr.e8ee4c2db44d.bib"},"thumbnail":{"src":"/content/papers/assets/fe0f3bb5164b182f59bf735f76d6487408377e22beebbfe6f11bf0f6caba5afd.jpg","width":1464,"height":432},"locale":"en","path":"/content/papers/en/text-recognition/diffusionstr.e31657d2a01f.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"solution","text":"Solution","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"transformer-architecture","text":"Transformer Architecture","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"comparison-with-other-methods","text":"Comparison with Other Methods","depth":3},{"id":"advantages-of-diffusion-models","text":"Advantages of Diffusion Models","depth":3},{"id":"impact-of-character-aware-head","text":"Impact of Character-Aware Head","depth":3},{"id":"effect-of-diffusion-steps","text":"Effect of Diffusion Steps","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"text-spotting/got","relation":"comparison","topic":"Generative approaches to OCR"},{"id":"text-recognition/dtrocr","relation":"comparison","topic":"Generative approaches to OCR"}]},{"id":"text-spotting/got","category":"text-spotting","legacyPath":"text-spotting/2409-got","legacySlug":"2409-got","legacyPaths":[],"title":"GOT","subtitle":"All-encompassing OCR","author":"Z. Yuan","paperDate":"2024-09-03","publishedOn":"2024-10-06","paperTitle":"General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model","paperUrl":"https://arxiv.org/abs/2409.01704","bibliography":{"title":"General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model","authors":["Haoran Wei","Chenglong Liu","Jinyue Chen","Jia Wang","Lingyu Kong","Yanming Xu","Zheng Ge","Liang Zhao","Jianjian Sun","Yuang Peng","Chunrui Han","Xiangyu Zhang"],"url":"https://arxiv.org/abs/2409.01704","pdfUrl":"https://arxiv.org/pdf/2409.01704","codeUrl":null,"preprint":{"id":"2409.01704","submittedOn":"2024-09-03","url":"https://arxiv.org/abs/2409.01704"},"publication":null,"citation":"@misc{textspottinggot2024,\n  title = {{General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model}},\n  author = {Haoran Wei and Chenglong Liu and Jinyue Chen and Jia Wang and Lingyu Kong and Yanming Xu and Zheng Ge and Liang Zhao and Jianjian Sun and Yuang Peng and Chunrui Han and Xiangyu Zhang},\n  year = {2024},\n  eprint = {2409.01704},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2409.01704}\n}\n","citationPath":"/content/papers/citations/text-spotting/got.a6192bad407a.bib"},"thumbnail":{"src":"/content/papers/assets/7ed02fbacb92b589ea5efc9c02a180ddc03e3af515428f8db5b3a64ab0b754dd.jpg","width":1224,"height":908},"locale":"en","path":"/content/papers/en/text-spotting/got.a7bec0775c96.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solving","text":"Problem Solving","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"phase-1-training","text":"Phase 1 Training","depth":3},{"id":"training-data","text":"Training Data","depth":4},{"id":"phase-2-training","text":"Phase 2 Training","depth":3},{"id":"phase-3-training","text":"Phase 3 Training","depth":3},{"id":"training-strategy","text":"Training Strategy","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"plain-text-document-ocr-performance","text":"Plain Text Document OCR Performance","depth":3},{"id":"scene-text-ocr-performance","text":"Scene Text OCR Performance","depth":3},{"id":"formatted-document-ocr-performance","text":"Formatted Document OCR Performance","depth":3},{"id":"fine-grained-ocr-performance","text":"Fine-grained OCR Performance","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{},"guide":null,"relatedReading":[{"id":"text-recognition/dtrocr","relation":"comparison","topic":"Generative approaches to OCR"},{"id":"text-recognition/diffusionstr","relation":"comparison","topic":"Generative approaches to OCR"}]},{"id":"text-recognition/dtrocr","category":"text-recognition","legacyPath":"text-recognition/2308-dtrocr","legacySlug":"2308-dtrocr","legacyPaths":[],"title":"DTrOCR","subtitle":"Decoder-Only Approach","author":"Z. Yuan","paperDate":"2023-08-30","publishedOn":"2024-11-15","paperTitle":"DTrOCR: Decoder-only Transformer for Optical Character Recognition","paperUrl":"https://arxiv.org/abs/2308.15996","bibliography":{"title":"DTrOCR: Decoder-only Transformer for Optical Character Recognition","authors":["Masato Fujitake"],"url":"https://arxiv.org/abs/2308.15996","pdfUrl":"https://arxiv.org/pdf/2308.15996","codeUrl":null,"preprint":{"id":"2308.15996","submittedOn":"2023-08-30","url":"https://arxiv.org/abs/2308.15996"},"publication":{"venue":"WACV","year":2024,"kind":"conference","url":"https://arxiv.org/abs/2308.15996"},"citation":"@misc{textrecognitiondtrocr2023,\n  title = {{DTrOCR: Decoder-only Transformer for Optical Character Recognition}},\n  author = {Masato Fujitake},\n  year = {2023},\n  eprint = {2308.15996},\n  archivePrefix = {arXiv},\n  url = {https://arxiv.org/abs/2308.15996}\n}\n","citationPath":"/content/papers/citations/text-recognition/dtrocr.dcb015457224.bib"},"thumbnail":{"src":"/content/papers/assets/192548a108c1789007d9a90f8ecb479c783d1b198a507d1742a4c8e5cbadc81a.jpg","width":1224,"height":660},"locale":"en","path":"/content/papers/en/text-recognition/dtrocr.f34ff9819780.md","headings":[{"id":"problem-definition","text":"Problem Definition","depth":2},{"id":"problem-solving","text":"Problem Solving","depth":2},{"id":"model-architecture","text":"Model Architecture","depth":3},{"id":"synthetic-dataset-pretraining","text":"Synthetic Dataset Pretraining","depth":3},{"id":"real-data-fine-tuning","text":"Real Data Fine-Tuning","depth":3},{"id":"implementation-details","text":"Implementation Details","depth":3},{"id":"discussion","text":"Discussion","depth":2},{"id":"str-comparison","text":"STR Comparison","depth":3},{"id":"sroie-comparison","text":"SROIE Comparison","depth":3},{"id":"handwritten-performance","text":"Handwritten Performance","depth":3},{"id":"ctr-performance","text":"CTR Performance","depth":3},{"id":"conclusion","text":"Conclusion","depth":2}],"anchorAliases":{"real-dataset-pretraining":"real-data-fine-tuning"},"guide":null,"relatedReading":[{"id":"text-recognition/trocr","relation":"background","topic":"Pretraining Transformers for text recognition"},{"id":"text-spotting/got","relation":"comparison","topic":"Generative approaches to OCR"},{"id":"text-recognition/diffusionstr","relation":"comparison","topic":"Generative approaches to OCR"}]}],"text":"Sometimes the logic of research is simple:\n\nSpot something promising in another field, try it on your own problem, and perhaps you have a paper.\n\n## Problem Definition\n\nDiffusion models have been popular for several years, mostly in image generation.\n\nSomething this new and interesting...\n\nWhy not try it on text recognition?\n\n## Solution\n\n### Model Architecture\n\n![ViT, text-sequence diffusion, FFN and character-aware head; original paper Figure 2](/content/papers/assets/fe0f3bb5164b182f59bf735f76d6487408377e22beebbfe6f11bf0f6caba5afd.jpg \"Fujitake, DiffusionSTR, Figure 2, arXiv:2306.16707v1\")\n\nWe could talk about diffusion models for days, so I will skip the basics and assume you are already an expert.\n\nHere the authors use a diffusion model as the basis for text recognition.\n\nThe architecture is quite simple. First, an image is encoded using a Vision Transformer (ViT), extracting visual feature representations. A sequence filled with noise, $x_T$, is fed into a Transformer, which iteratively refines this noisy sequence into $x_{T-1}$, repeating this process $T$ times, until the final output sequence, $x_0$, is obtained.\n\nIn the last step, $x_0$ is passed through a feedforward neural network (FFN) to produce the recognized text sequence, while a character-aware head predicts which positions in the fixed-length sequence contain characters. These are sequence positions, not character boxes in the image.\n\nAs background, a continuous diffusion model has two directions:\n\n- **Forward Process**: Adds noise step by step to the data point $x_0$ until it becomes pure Gaussian noise, $x_T$.\n- **Reverse Process**: Gradually denoises this noisy signal to reconstruct the original data $x_0$.\n\nThe authors use multinomial diffusion for discrete characters, with special tokens such as [EOS] (end of sequence), [PAD] (padding), and [MASK] (noise). Thus the Gaussian process above describes the original diffusion model, not the categorical noise used for these tokens.\n\nTraining uses the simplified loss $L_{\\text{simple}}$ with mean squared error (MSE) for stability.\n\nThe authors also add no separate language model. They felt that a language-model branch would complicate the architecture and make the effect of the diffusion model harder to see.\n\n### Transformer Architecture\n\nThe authors leveraged a Transformer decoder with time-position encoding to transform sequences in the diffusion model.\n\nKey differences from traditional Transformers include:\n\n1. Introducing visual features $z$ to conditionally process the text sequence through a cross-attention mechanism.\n2. Predicting all character probabilities simultaneously rather than one at a time (non-autoregressive).\n3. The output has dual purposes: being converted into a text sequence via FFN and fed into a character-aware head for predicting whether each character position contains a real character.\n\nDuring training, a random timestep $t$ is sampled. The noise schedule $\\alpha_t, \\bar{\\alpha}_t$ is used to form the noisy sequence $x_t$, and the decoder predicts $x_{t-1} = \\text{Dec}(x_t, z, t)$ to calculate the loss.\n\nFor inference, the process begins with a fully masked (noisy) sequence $x_T$, and over $T$ iterations, conditioned on the visual features $z$ generated by the visual encoder, the model eventually outputs $x_0$. This final output is passed through FFN to produce the recognized text sequence.\n\nThe authors also introduced a character-aware head to classify whether positions in the fixed-length sequence correspond to real characters. This binary classification task uses a binary cross-entropy loss function.\n\n## Discussion\n\n### Comparison with Other Methods\n\n![Eight reported text-recognition test columns after MJ and ST training; original paper Table 1](/content/papers/assets/5eac1415202bd2bf9ddc5d933a205eac49611c274c2b9fefc6c02aed9c7f6d8c.jpg \"Fujitake, DiffusionSTR, Table 1, arXiv:2306.16707v1; IC13 and IC15 each have two test splits\")\n\nThe table compares word accuracy for methods trained only on MJ and ST across six test datasets. IC13 and IC15 each have two reported splits, giving eight result columns.\n\nDiffusionSTR is competitive with recent strong methods, though it does not lead in every column. Against PARSeq, it scores **97.3/97.0** on IIIT5k and ties at **93.6** on SVT, but scores **86.0/86.5** and **82.2/82.9** on the two IC15 splits.\n\nFor instance, ABINet and PARSeq use strong language models, while TRBA adopts an image correction module as a preprocessing step. Despite its simpler architecture without these additional techniques, DiffusionSTR achieves similar levels of accuracy.\n\n### Advantages of Diffusion Models\n\n\n\n![Recognition examples for DiffusionSTR and PARSeq on three images; original paper Figure 3](/content/papers/assets/709866602e96ac73b0b4c381aec91c9ca3f4106df811da6bc346a4ca70d21eab.jpg \"Fujitake, DiffusionSTR, Figure 3, arXiv:2306.16707v1; black text matches the label, red text differs\")\n\n\n\nThe authors show three recognition examples, including noisy and blurred or curved text. Black text matches the ground truth; red text differs. These examples illustrate particular cases, rather than measuring robustness across noise or blur levels:\n\n1. **Noisy image:** DiffusionSTR gets closer to the text in one pictured case than PARSeq.\n2. **Blurred image:** It can still recognize some text in the pictured examples.\n\n### Impact of Character-Aware Head\n\n![Average accuracy with and without the character-aware head; original paper Table 2](/content/papers/assets/043d25437e44bac4753fb847f6dcd3840325717dc9d52757506abffdd67598c8.jpg \"Fujitake, DiffusionSTR, Table 2, arXiv:2306.16707v1\")\n\nRemoving the character-aware head lowers average accuracy across the six test datasets from **91.8%** to **63.4%**. In this fixed-length sequence design, predicting which positions contain characters matters a great deal.\n\n### Effect of Diffusion Steps\n\n![Average accuracy from 100 to 4000 diffusion steps; original paper Table 3](/content/papers/assets/40aca308329f6eb7f28f4bcb352710bb01c8248e95560c99c6deb78f067c23b4.jpg \"Fujitake, DiffusionSTR, Table 3, arXiv:2306.16707v1\")\n\nThe table shows how the total number of steps affects average accuracy: it rises from **17.6%** at 100 steps to **91.9%** at 2000 steps, then falls slightly to **91.3%** at 4000 steps.\n\nAfter 1000 steps, accuracy in this experiment is roughly saturated.\n\n## Conclusion\n\nThe method uses visual information to refine a noisy text sequence step by step, taking a different route from methods that map an image directly to text. On the reported test sets, DiffusionSTR is competitive with leading methods. That is a good demonstration that the idea works.\n\nThis new way of thinking shows researchers what diffusion models might offer text recognition.\n\n\n> [!TIP]\n> \n> The authors say nothing about inference time in this paper.\n> \n> Since the diffusion model has to iterate repeatedly, I suspect this one may be fairly slow.\n","imageSizes":{"/content/papers/assets/fe0f3bb5164b182f59bf735f76d6487408377e22beebbfe6f11bf0f6caba5afd.jpg":{"width":1464,"height":432},"/content/papers/assets/5eac1415202bd2bf9ddc5d933a205eac49611c274c2b9fefc6c02aed9c7f6d8c.jpg":{"width":1464,"height":486},"/content/papers/assets/709866602e96ac73b0b4c381aec91c9ca3f4106df811da6bc346a4ca70d21eab.jpg":{"width":824,"height":464},"/content/papers/assets/043d25437e44bac4753fb847f6dcd3840325717dc9d52757506abffdd67598c8.jpg":{"width":1114,"height":258},"/content/papers/assets/40aca308329f6eb7f28f4bcb352710bb01c8248e95560c99c6deb78f067c23b4.jpg":{"width":1120,"height":168}}}
