Back to paper library
arXiv 2019ICCV 2019Text recognitionPaper notes

WWWSTR

Data and Model Analysis

Original paper

What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis

Paper authors

Jeonghun Baek · Geewook Kim · Junyeop Lee …

Show all 8 authors

Jeonghun Baek · Geewook Kim · Junyeop Lee · Sungrae Park · Dongyoon Han · Sangdoo Yun · Seong Joon Oh · Hwalsuk Lee

Publication
ICCV 2019Conference
First submitted to arXiv
arXiv:1904.01906
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionwwwstr2019,
  title = {{What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis}},
  author = {Jeonghun Baek and Geewook Kim and Junyeop Lee and Sungrae Park and Dongyoon Han and Sangdoo Yun and Seong Joon Oh and Hwalsuk Lee},
  year = {2019},
  eprint = {1904.01906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.01906}
}
On this page

Clova AI studies existing datasets and model components within a unified comparison framework. Its contribution is a controlled analysis of training data and module combinations, including combinations not previously explored.

Dataset Analysis

Different studies have used various dataset combinations for training, which may result in unclear performance improvements. It becomes difficult to determine if the improvement is due to the proposed new model or the usage of a better or larger dataset.

This inconsistency in datasets affects the fairness of results.

Training Datasets

Due to the high cost of annotating scene text images, it's challenging to acquire sufficient labeled data, and most STR (Scene Text Recognition) models opt for synthetic datasets for training.

Below are two widely used synthetic datasets in STR research:

  • MJSynth (MJ)

    • Synthetic Word Dataset

      MJSynth is a synthetic dataset specifically designed for STR, containing 8.9 million word-box images. The generation process involves font rendering, border and shadow rendering, background coloring, merging fonts and backgrounds, applying perspective distortions, mixing with real images, and adding noise.

  • SynthText (ST)

    • SynthText in the Wild Dataset

      SynthText was originally created for scene text detection but has also been used in STR research. Researchers cropped word boxes and filtered out samples containing non-alphanumeric characters for STR training, leaving about 5.5 million training images.

Testing Datasets

Different versions of benchmark datasets have certain differences.

For example, in IC03, a difference of 7 samples can lead to a 0.8-percentage-point accuracy gap, which is a significant difference when comparing prior research results.

The sample number differences in IC13 and IC15 are even larger than those in IC03.

Seven real-scene STR datasets are widely used for post-training model evaluation. These datasets can be categorized into "regular datasets" and "irregular datasets" based on the difficulty of the text and its geometric layout:

  • Regular Datasets: These contain text images where characters are evenly spaced and arranged horizontally, making them relatively easier to process:

    • IIIT5K-Words (IIIT): A dataset from Google image search containing 2,000 training images and 3,000 evaluation images.
    • Street View Text (SVT): Collected from Google Street View, this dataset contains outdoor street scenes, with some images affected by noise, blur, or low resolution. It includes 257 training images and 647 evaluation images.
    • ICDAR2003 (IC03): Used in the ICDAR 2003 Robust Reading competition, it contains 1,156 training images and 1,110 evaluation images. There are two versions (860 and 867 images) due to the exclusion of words that are too short or contain non-alphanumeric characters.
    • ICDAR2013 (IC13): This dataset inherits most of the images from IC03 and was created for the ICDAR 2013 Robust Reading competition. It contains 848 training images and 1,095 evaluation images. Two versions are available for evaluation, one with 857 images and the other with 1,015.

  • Irregular Datasets: These contain more challenging scene text, such as curved, rotated, or distorted text:

    • ICDAR2015 (IC15): This dataset, used in the ICDAR 2015 Robust Reading competition, contains 4,468 training images and 2,077 evaluation images. The images are captured from a natural movement perspective using Google Glass, resulting in many images with noise, blur, and rotation. Evaluation versions include 1,811 and 2,077 images.
    • SVT Perspective (SP): Collected from Google Street View, it includes 645 evaluation images, many with perspective distortions due to non-frontal views.
    • CUTE80 (CT): A dataset collected from natural scenes, containing 288 cropped evaluation images, many of which feature curved text.

Version Differences

Based on the table above, different studies have used different versions of benchmark datasets for model evaluation, particularly in the IC03, IC13, and IC15 datasets.

For example, in IC03, a difference of 7 samples can lead to a 0.8-percentage-point accuracy gap, which is significant when comparing results. The differences in sample sizes between IC13 and IC15 are even larger. Such versions can bias cross-paper comparisons, so they need to be checked before comparing models.

Architecture Analysis

STR converts a cropped text image into a character sequence, combining visual feature extraction with sequence modeling. This comparison concerns recognition, rather than the detection stage that locates text boxes.

CRNN is a representative model combining CNN and RNN for STR, which used CNNs for feature extraction and RNNs for sequence prediction. Many variants have since been proposed to improve performance, introducing different modules to handle complex features like font styles and backgrounds.

Some methods even omitted the RNN stage to reduce inference time. Subsequent research has introduced attention-based decoders to improve character sequence prediction.

The STR architecture typically follows these four stages:

  1. Transformation: Spatial transformation networks (STN) are used to normalize input text images, reducing the burden on subsequent stages.
  2. Feature Extraction: This stage converts input images into representations focused on character recognition while reducing interference from changes in font, color, size, and background.
  3. Sequence Modeling: This stage captures contextual information in the character sequence, improving the accuracy of each character's prediction.
  4. Prediction: The final character sequence is predicted based on the extracted features.

Transformation Stage

This stage transforms input images X into normalized images X'.

Since text images in natural scenes vary in shape (such as curved or slanted text), if passed directly to later stages, the feature extraction stage would need to learn representations invariant to these geometric distortions.

To alleviate this burden, some research has employed Thin-Plate Spline (TPS) transformation, a variant of spatial transformation networks (STN). It predicts fiducial points along the upper and lower text boundaries, then uses smooth spline interpolation to normalize the text region.

TPS is flexible in handling varying scales of text images and standardizes text regions into predetermined rectangles.

Feature Extraction Stage

In this stage, CNNs abstract the input image (X or X') into visual feature maps.

Each column of the feature map corresponds to a receptive field in the input image. These fields may overlap and need not align with individual characters.

Three common architectures are studied: VGG, RCNN, and ResNet:

  • VGG: The convolutional feature extractor is used here, without its image-classification fully connected head.
  • RCNN: A recurrent CNN that reuses convolutional operations; this is not the region-based R-CNN used in object detection.
  • ResNet: Incorporates residual connections to alleviate the difficulties of training deep CNNs.

Sequence Modeling Stage

The output from the feature extraction stage is reshaped into a feature sequence V, where each column vi∈Vv_i \in V is treated as one frame in the sequence.

However, this sequence may lack contextual information, so previous studies have used Bidirectional Long Short-Term Memory (BiLSTM) networks to enhance the sequence H=Seq.(V)H = Seq.(V), capturing contextual relationships in the character sequence.

Prediction Stage

In this stage, the character sequence Y=y1,y2,…Y = y_1, y_2, … is predicted from sequence HH.

Based on previous research, the authors provide two prediction options:

  1. Connectionist Temporal Classification (CTC): This method predicts sequences of variable length with a fixed number of features. Decoding first merges adjacent repeated labels, then removes blanks. Identical characters separated by a blank remain distinct.
  2. Attention-based Sequence Prediction (Attn): This method automatically captures the information flow in the input sequence, learning a character-level language model that represents the dependencies between output classes.

These modules can be selected or adjusted as needed to suit different STR applications.

Experimental Setup

As previously mentioned, the dataset significantly affects STR model performance. To ensure a fair comparison, the authors standardized the training, validation, and evaluation datasets used.

  • Training Dataset: A combined dataset of MJSynth (8.9 million samples) and SynthText (5.5 million samples), totaling 14.4 million samples.
  • Validation Dataset: A combination of the training sets from IC13, IC15, IIIT, and SVT, used as the validation dataset.
  • Training Parameters: AdaDelta optimizer, decay rate ρ=0.95, batch size 192, total iterations 300,000, gradient clipping at 5, He initialization method, and model validation every 2,000 steps.
  • Duplicate Data Handling: The IC03 training set was excluded from validation, because 34 of its scene images (containing 215 word boxes) overlap the IC13 evaluation set.
  • Evaluation Metrics:
    • Accuracy: Word recognition accuracy across versions of seven real-world datasets and the combined 8,539-image benchmark; each module combination was trained with five random seeds and averaged.
    • Speed: Average processing time per image (in milliseconds).
    • Model size: The trainable parameter count serves as a storage-cost proxy, not a measurement of peak inference memory.
  • Experimental Environment: Intel Xeon(R) E5-2630 v4 2.20GHz CPU, NVIDIA TESLA P40 GPU, 252GB RAM.

Discussion

The authors analyzed the accuracy-speed and accuracy-memory trade-offs for different module combinations.

The graph below shows the trade-off curves for all module combinations, including six previously proposed STR models (marked with asterisks).

Accuracy-Time Trade-off Analysis

According to chart (a), T1 is the fastest model as it does not include any transformation or sequence modules.

From T1 to T5, the modules are progressively added (in bold): ResNet, BiLSTM, TPS (Thin-Plate Spline), and Attn (Attention mechanism).

Through these changes from T1 to T5, only one module is altered each time, providing a smooth transition between performance and computational efficiency, allowing minimal trade-offs between performance and efficiency depending on the application scenario.

ResNet, BiLSTM, and TPS significantly improve accuracy (69.5%→82.9%) with a moderate speed drop overall (1.3 ms → 10.9 ms).

Attn adds another 1.1 percentage points, while time per image rises from 10.9 ms to 27.6 ms—a substantial time cost.

Accuracy-Memory Trade-off Analysis

Chart (b) uses parameter count as its memory proxy. P1 has the fewest parameters, and the P1–P5 sequence illustrates the accuracy–model-size trade-off.

In the accuracy–parameter trade-off, one module changes at a time, with accuracy improving as the sequence adds Attn, TPS, BiLSTM, then ResNet. From P4 to P5, parameters jump from 7.2M to 49.6M, yet inference time actually drops from 30.1 ms to 27.6 ms.

Compared to VGG used in T1, RCNN is more lightweight in P1-P4, offering good accuracy-memory trade-offs. RCNN uses fewer standalone CNN layers, which are recursively applied, providing lightweight and efficient performance.

Within the P1–P4 comparison, transformation, sequence, and prediction modules raise parameter count from 1.9M to 7.2M and accuracy from 75.4% to 82.3%. This parameter increase is smaller than the subsequent switch to this paper's ResNet configuration.

ResNet raises accuracy by 1.7 percentage points in the final step, bringing the parameter count to 49.6M. Applications with a tight storage budget should evaluate this specific configuration.

Module Analysis

The authors analyzed the modules by accuracy, speed, and parameter count. The table's accuracy averages all combinations containing a module; its time and parameter figures are individual module measurements.

  • Accuracy Improvement: Compared to regular benchmark datasets, performance improvements on irregular datasets are approximately double. When comparing accuracy improvement and time usage, the optimal module upgrade order is: ResNet, BiLSTM, TPS, Attn. This is the most effective order for upgrading from the baseline combination (None-VGG-None-CTC) and aligns with the accuracy-time trade-off upgrade sequence (T1→T5).
  • Accuracy-Memory Perspective: With parameter count as the proxy, the efficient upgrade sequence is RCNN, Attn, TPS, BiLSTM, ResNet, matching P1→P5. After the initial RCNN choice, the remaining four upgrades run in the opposite order to the time-frontier sequence.

Failure Case Analysis

The authors also analyzed failures shared by the 24 module combinations.

The chart shows six common types of failures. Of the 8,539 examples in the benchmark datasets, 644 images (7.5%) were not correctly recognized by any model.

  1. Calligraphy Fonts:

    • Special fonts for brands (e.g., "Coca Cola") or street store names (e.g., "Cafe") remain a challenge. These diverse font styles require a new feature extractor to provide more adaptable visual features.
    • Future Research: Regularization may help prevent the model from overfitting to the font styles in the training dataset.
  2. Vertical Text:

    • Most current STR models assume that the text images are oriented horizontally, making it difficult to handle vertical text. Some models have attempted to utilize vertical information, but handling vertical text remains underdeveloped.
    • Future Research: Further research may need to focus on recognizing vertical text more effectively.
  3. Special Characters:

    • Since current benchmark tests do not evaluate special characters, existing research excludes them during training, causing models to misclassify these characters as alphabetic or numeric.
    • Experimental evidence: Including special characters raised IIIT accuracy from 87.9% to 90.3% in the authors’ setup. It motivates revisiting the character set, without guaranteeing the same gain for other models.
  4. Severe Occlusion:

    • Current methods do not fully utilize contextual information to overcome occlusion issues.
    • Future Research: Using stronger language models to maximize the use of contextual information could help.
  5. Low Resolution:

    • Existing models do not explicitly address low-resolution cases.
    • Future Research: Using image pyramids or super-resolution modules may improve performance.
  6. Label Noise:

    • Some failure cases are due to mislabeled data in the datasets. Upon examination, all benchmark datasets contain noisy labels.
    • Label Noise Statistics:
      • The proportion of incorrect labels without considering special characters is 1.3%.
      • The proportion of incorrect labels considering special characters is 6.1%.
      • The proportion of incorrect labels considering case sensitivity is 24.1%.

Conclusion

This paper provides an in-depth and comprehensive analysis of the scene text recognition field, offering a unified framework for fairly comparing the performance of different models.

If you don’t have time to read other papers, at least read this one!

It will help you quickly understand the state of the text recognition field as of 2019 and provide clear directions for your research, avoiding many potential pitfalls.