Back to paper library
arXiv 2023ICIP 2023Text recognitionPaper notes

DiffusionSTR

Text Signals Amidst Noise

Original paper

DiffusionSTR: Diffusion Model for Scene Text Recognition

Paper authors

Masato Fujitake

Publication
ICIP 2023Conference
First submitted to arXiv
arXiv:2306.16707
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitiondiffusionstr2023,
  title = {{DiffusionSTR: Diffusion Model for Scene Text Recognition}},
  author = {Masato Fujitake},
  year = {2023},
  eprint = {2306.16707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.16707}
}
On this page

Sometimes the logic of research is simple:

Spot something promising in another field, try it on your own problem, and perhaps you have a paper.

Problem Definition

Diffusion models have been popular for several years, mostly in image generation.

Something this new and interesting...

Why not try it on text recognition?

Solution

Model Architecture

Fujitake, DiffusionSTR, Figure 2, arXiv:2306.16707v1

We could talk about diffusion models for days, so I will skip the basics and assume you are already an expert.

Here the authors use a diffusion model as the basis for text recognition.

The architecture is quite simple. First, an image is encoded using a Vision Transformer (ViT), extracting visual feature representations. A sequence filled with noise, xTx_T, is fed into a Transformer, which iteratively refines this noisy sequence into xT−1x_{T-1}, repeating this process TT times, until the final output sequence, x0x_0, is obtained.

In the last step, x0x_0 is passed through a feedforward neural network (FFN) to produce the recognized text sequence, while a character-aware head predicts which positions in the fixed-length sequence contain characters. These are sequence positions, not character boxes in the image.

As background, a continuous diffusion model has two directions:

  • Forward Process: Adds noise step by step to the data point x0x_0 until it becomes pure Gaussian noise, xTx_T.
  • Reverse Process: Gradually denoises this noisy signal to reconstruct the original data x0x_0.

The authors use multinomial diffusion for discrete characters, with special tokens such as [EOS] (end of sequence), [PAD] (padding), and [MASK] (noise). Thus the Gaussian process above describes the original diffusion model, not the categorical noise used for these tokens.

Training uses the simplified loss LsimpleL_{\text{simple}} with mean squared error (MSE) for stability.

The authors also add no separate language model. They felt that a language-model branch would complicate the architecture and make the effect of the diffusion model harder to see.

Transformer Architecture

The authors leveraged a Transformer decoder with time-position encoding to transform sequences in the diffusion model.

Key differences from traditional Transformers include:

  1. Introducing visual features zz to conditionally process the text sequence through a cross-attention mechanism.
  2. Predicting all character probabilities simultaneously rather than one at a time (non-autoregressive).
  3. The output has dual purposes: being converted into a text sequence via FFN and fed into a character-aware head for predicting whether each character position contains a real character.

During training, a random timestep tt is sampled. The noise schedule αt,αˉt\alpha_t, \bar{\alpha}_t is used to form the noisy sequence xtx_t, and the decoder predicts xt−1=Dec(xt,z,t)x_{t-1} = \text{Dec}(x_t, z, t) to calculate the loss.

For inference, the process begins with a fully masked (noisy) sequence xTx_T, and over TT iterations, conditioned on the visual features zz generated by the visual encoder, the model eventually outputs x0x_0. This final output is passed through FFN to produce the recognized text sequence.

The authors also introduced a character-aware head to classify whether positions in the fixed-length sequence correspond to real characters. This binary classification task uses a binary cross-entropy loss function.

Discussion

Comparison with Other Methods

Fujitake, DiffusionSTR, Table 1, arXiv:2306.16707v1; IC13 and IC15 each have two test splits

The table compares word accuracy for methods trained only on MJ and ST across six test datasets. IC13 and IC15 each have two reported splits, giving eight result columns.

DiffusionSTR is competitive with recent strong methods, though it does not lead in every column. Against PARSeq, it scores 97.3/97.0 on IIIT5k and ties at 93.6 on SVT, but scores 86.0/86.5 and 82.2/82.9 on the two IC15 splits.

For instance, ABINet and PARSeq use strong language models, while TRBA adopts an image correction module as a preprocessing step. Despite its simpler architecture without these additional techniques, DiffusionSTR achieves similar levels of accuracy.

Advantages of Diffusion Models

Fujitake, DiffusionSTR, Figure 3, arXiv:2306.16707v1; black text matches the label, red text differs

The authors show three recognition examples, including noisy and blurred or curved text. Black text matches the ground truth; red text differs. These examples illustrate particular cases, rather than measuring robustness across noise or blur levels:

  1. Noisy image: DiffusionSTR gets closer to the text in one pictured case than PARSeq.
  2. Blurred image: It can still recognize some text in the pictured examples.

Impact of Character-Aware Head

Fujitake, DiffusionSTR, Table 2, arXiv:2306.16707v1

Removing the character-aware head lowers average accuracy across the six test datasets from 91.8% to 63.4%. In this fixed-length sequence design, predicting which positions contain characters matters a great deal.

Effect of Diffusion Steps

Fujitake, DiffusionSTR, Table 3, arXiv:2306.16707v1

The table shows how the total number of steps affects average accuracy: it rises from 17.6% at 100 steps to 91.9% at 2000 steps, then falls slightly to 91.3% at 4000 steps.

After 1000 steps, accuracy in this experiment is roughly saturated.

Conclusion

The method uses visual information to refine a noisy text sequence step by step, taking a different route from methods that map an image directly to text. On the reported test sets, DiffusionSTR is competitive with leading methods. That is a good demonstration that the idea works.

This new way of thinking shows researchers what diffusion models might offer text recognition.