Back to paper library
arXiv 2021Text recognitionPaper notes

YATR

Fake Can't Replace the Real

Original paper

Why You Should Try the Real Data for the Scene Text Recognition

Paper authors

Vladimir Loginov

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2107.13938
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionyatr2021,
  title = {{Why You Should Try the Real Data for the Scene Text Recognition}},
  author = {Vladimir Loginov},
  year = {2021},
  eprint = {2107.13938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.13938}
}
On this page

In the field of Scene Text Recognition (STR), the majority of models rely heavily on synthetic data for training.

The standard approach is to begin with pre-training on synthetic data, and then either evaluate the model directly on benchmark test sets or fine-tune it using a small amount of real data before proceeding with evaluation.

Defining the Problem

There’s no denying the importance of real-world data. Synthetic data must be generated based on specific rules—such as selecting particular fonts, adding noise, or choosing backgrounds.

Where there are rules, there are exceptions. These exceptions are often things that synthetic data cannot cover, which can impact the model’s ability to generalize. In the real world, synthetic data is the exception; real data is the rule.

Would you trust a model's generalization ability if it’s trained on exceptions instead of the rule?

The authors believe that combining synthetic and real data makes a better training set, and they test that idea here.

Solving the Problem

Model Architecture

Loginov, Why You Should Try the Real Data for the Scene Text Recognition, Figure 1, arXiv:2107.13938v1

The authors reference the approach used in WWWSTR:

However, they do not separate the last two stages; in this architecture, the final stage of text recognition incorporates both sequence modeling and the final prediction in a single step.

For image rectification, the model employs the Thin Plate Spline (TPS) technique from ASTER:

The TPS spatial transformation mechanism consists of three components:

  1. Localization Network: First, it generates the spatial transformation parameters for the input image.
  2. Grid Generator: Based on the predicted parameters, it creates a set of sampling grid points marking where the input image should be sampled.
  3. Sampler: Using the grid and the input image, it generates the final output image.

The strength of TPS lies in its design, which doesn’t require character-level annotation, making it adaptable to various text recognition architectures. Its parameters can be learned in an end-to-end manner.

The backbone uses ImageNet-pretrained ResNeXt-101 without its final stage. The authors chose this configuration for its favorable accuracy and efficiency trade-off in their experiments.

Finally, for the text recognition head, a 2D attention map with a GRU decoder is employed.

Discussion

The Importance of Real Data

Loginov, Table 2, arXiv:2107.13938v1; this ablation did not use TPS

The authors conducted training experiments using both synthetic and real datasets, keeping all conditions consistent: ResNeXt-101 as the backbone network, a case-insensitive training and testing mode, and no use of the TPS module in this experiment.

In this no-TPS comparison, MJSynth plus OpenImagesV5 scores higher than either synthetic-only setup on all six test sets. OpenImagesV5 alone scores below that mixture but still above MJSynth alone on all six. For example, on IC15 the scores are 64.2% with MJSynth, 69.3% with MJSynth plus SynthText, and 75.3% with MJSynth plus OpenImagesV5. The authors suggest that harder real images might need a longer training schedule, but the table does not test training duration.

MJSynth + OpenImagesV5 gives the best results in this table and is the training-data mixture the authors use in this paper.

Testing with ViTSTR

To further validate the importance of real data, the authors experimented with ViTSTR, a Transformer-based text recognition model.

You can refer to our earlier discussion of ViTSTR:

Using both synthetic and real datasets for training, they obtained the following results:

Loginov, Table 4, arXiv:2107.13938v1; both setups used augmentation

Table 4 compares two ViTSTR-Tiny runs under the same test conditions and with data augmentation in both. Replacing 20% of a 0.5 MJSynth + 0.5 SynthText training batch with OpenImagesV5 raises accuracy on all six test sets. IC03 goes from 93.7% to 95.6% (1.9 percentage points), while IC15 goes from 66.6% to 77.0% (10.4 points); the gain is not roughly five points everywhere. Even with image augmentation, adding real data helps in this comparison.

Conclusion

Combining synthetic and real images improved text recognition on the models and test sets examined here.

You might think this is common sense and hardly needs another paper. But without measuring, how would we know how much it matters?

The next time someone asks, “Why use real data? What is the actual difference?”, this paper gives us numbers to answer with.