DiffusionSTR: Diffusion Model for Scene Text Recognition
- Publication
- ICIP 2023Conference
- First submitted to arXiv
- arXiv:2306.16707
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textrecognitiondiffusionstr2023,
title = {{DiffusionSTR: Diffusion Model for Scene Text Recognition}},
author = {Masato Fujitake},
year = {2023},
eprint = {2306.16707},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2306.16707}
}
Sometimes the logic of research is simple:
Spot something promising in another field, try it on your own problem, and perhaps you have a paper.
Problem Definition
Diffusion models have been popular for several years, mostly in image generation.
Something this new and interesting...
Why not try it on text recognition?
Solution
Model Architecture
We could talk about diffusion models for days, so I will skip the basics and assume you are already an expert.
Here the authors use a diffusion model as the basis for text recognition.
The architecture is quite simple. First, an image is encoded using a Vision Transformer (ViT), extracting visual feature representations. A sequence filled with noise, , is fed into a Transformer, which iteratively refines this noisy sequence into , repeating this process times, until the final output sequence, , is obtained.
In the last step, is passed through a feedforward neural network (FFN) to produce the recognized text sequence, while a character-aware head predicts which positions in the fixed-length sequence contain characters. These are sequence positions, not character boxes in the image.
As background, a continuous diffusion model has two directions:
- Forward Process: Adds noise step by step to the data point until it becomes pure Gaussian noise, .
- Reverse Process: Gradually denoises this noisy signal to reconstruct the original data .
The authors use multinomial diffusion for discrete characters, with special tokens such as [EOS] (end of sequence), [PAD] (padding), and [MASK] (noise). Thus the Gaussian process above describes the original diffusion model, not the categorical noise used for these tokens.
Training uses the simplified loss with mean squared error (MSE) for stability.
The authors also add no separate language model. They felt that a language-model branch would complicate the architecture and make the effect of the diffusion model harder to see.
Transformer Architecture
The authors leveraged a Transformer decoder with time-position encoding to transform sequences in the diffusion model.
Key differences from traditional Transformers include:
- Introducing visual features to conditionally process the text sequence through a cross-attention mechanism.
- Predicting all character probabilities simultaneously rather than one at a time (non-autoregressive).
- The output has dual purposes: being converted into a text sequence via FFN and fed into a character-aware head for predicting whether each character position contains a real character.
During training, a random timestep is sampled. The noise schedule is used to form the noisy sequence , and the decoder predicts to calculate the loss.
For inference, the process begins with a fully masked (noisy) sequence , and over iterations, conditioned on the visual features generated by the visual encoder, the model eventually outputs . This final output is passed through FFN to produce the recognized text sequence.
The authors also introduced a character-aware head to classify whether positions in the fixed-length sequence correspond to real characters. This binary classification task uses a binary cross-entropy loss function.
Discussion
Comparison with Other Methods
The table compares word accuracy for methods trained only on MJ and ST across six test datasets. IC13 and IC15 each have two reported splits, giving eight result columns.
DiffusionSTR is competitive with recent strong methods, though it does not lead in every column. Against PARSeq, it scores 97.3/97.0 on IIIT5k and ties at 93.6 on SVT, but scores 86.0/86.5 and 82.2/82.9 on the two IC15 splits.
For instance, ABINet and PARSeq use strong language models, while TRBA adopts an image correction module as a preprocessing step. Despite its simpler architecture without these additional techniques, DiffusionSTR achieves similar levels of accuracy.
Advantages of Diffusion Models
The authors show three recognition examples, including noisy and blurred or curved text. Black text matches the ground truth; red text differs. These examples illustrate particular cases, rather than measuring robustness across noise or blur levels:
- Noisy image: DiffusionSTR gets closer to the text in one pictured case than PARSeq.
- Blurred image: It can still recognize some text in the pictured examples.
Impact of Character-Aware Head
Removing the character-aware head lowers average accuracy across the six test datasets from 91.8% to 63.4%. In this fixed-length sequence design, predicting which positions contain characters matters a great deal.
Effect of Diffusion Steps
The table shows how the total number of steps affects average accuracy: it rises from 17.6% at 100 steps to 91.9% at 2000 steps, then falls slightly to 91.3% at 4000 steps.
After 1000 steps, accuracy in this experiment is roughly saturated.
Conclusion
The method uses visual information to refine a noisy text sequence step by step, taking a different route from methods that map an image directly to text. On the reported test sets, DiffusionSTR is competitive with leading methods. That is a good demonstration that the idea works.
This new way of thinking shows researchers what diffusion models might offer text recognition.