Back to paper library
arXiv 2023ICCV 2023Text recognitionPaper notes

Union14M

Chess Pieces Falling Like Stars

Original paper

Revisiting Scene Text Recognition: A Data Perspective

Paper authors

Qing Jiang · Jiapeng Wang · Dezhi Peng · Chongyu Liu · Lianwen Jin

Publication
ICCV 2023Conference
First submitted to arXiv
arXiv:2307.08723
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionunion14m2023,
  title = {{Revisiting Scene Text Recognition: A Data Perspective}},
  author = {Qing Jiang and Jiapeng Wang and Dezhi Peng and Chongyu Liu and Lianwen Jin},
  year = {2023},
  eprint = {2307.08723},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.08723}
}
On this page

We’ve explored dozens of papers on scene text recognition (STR), and the benchmark test datasets in use tend to revolve around the same few: ICDAR, IIIT5K, SVT, SVTP, and CUTE80.

Truth be told, performance on these datasets is already near saturation.

Defining the Problem

As shown above, this graph represents the average performance across six widely-used STR datasets.

From CRNN through SAR, SATRN, and ABINet, the plotted results exceeded 92% average accuracy by 2022. Gains on these six benchmarks had become smaller.

Does this imply that STR is a “solved” problem? Or is it possible that the benchmark test datasets we rely on are too limited, preventing us from exposing deeper challenges within the field?

If these benchmarks are no longer enough, why not set them aside? The authors decide to overturn the chessboard and start again.

Problem Solution

Since current datasets no longer capture the full scope of real-world challenges in STR, the authors propose Union14M, a large-scale, unified dataset.

Union14M brings together 4 million labeled images (Union14M-L) and 10 million unlabeled images (Union14M-U), consolidated from 17 publicly available datasets. The structure is shown below:

By aggregating existing datasets and categorizing diverse challenges, Union14M aims to better represent the variability of real-world text, offering new insights into the limitations of current STR models and pushing the field forward.

Union14M-L

Union14M-L consists of 4 million labeled images gathered from 14 public datasets with the goal of representing diverse real-world scenarios. Some of these datasets include:

  • ArT: Focuses on curved text images.
  • ReCTS, RCTW, LSVT, KAIST, NEOCR, and IIIT-ILST: Provide street-view text from various countries.
  • MTWI: Extracts text images from web pages.
  • COCOTextV2: Includes low-resolution and vertical text images.
  • IntelOCR, TextOCR, and HierText: Draw images from OpenImages. The roughly 9 million images refer to OpenImages as a whole, not to the combined text annotations in these three datasets.

Simply merging these datasets, however, would not suffice. Challenges such as inconsistent annotations, duplicate samples, non-Latin characters, or corrupted images must be addressed. The authors applied several optimization strategies to ensure high-quality consolidation:

  • Cropping Text Instances: Instead of cropping along polygon annotations, the authors used the minimum axis-aligned rectangle around each text instance. This can include more background noise, challenging the model during training and helping it become more robust to such noise. It also reduces reliance on highly precise detection crops when analyzing recognition performance.

  • Removing Duplicates: To eliminate redundancy, the authors first filtered out overlaps between Union14M-L and other common benchmarks. They also removed duplicate samples within the 14 datasets. For example, HierText, TextOCR, and IntelOCR overlap, as they all derive from OpenImages. The authors kept HierText as the main reference, removing the redundant samples from the other two datasets.

  • Excluding Non-Latin Characters and Ignored Samples: To focus on Latin-based text, only samples containing letters, numbers, or symbols were kept, while images marked as "ignored" were excluded.

Union14M-U

Union14M-U comprises 10 million unlabeled images, leveraging self-supervised learning—a growing trend in computer vision and STR. Since manually labeling text images is both time-consuming and requires linguistic expertise, the authors explored how unlabeled data can enhance STR models.

They sourced these unlabeled images from Book32, OpenImages, and Conceptual Captions (CC). To select high-quality text instances, the authors employed three different text detectors and adopted an IoU voting mechanism to ensure consistency. Additionally, they removed duplicate samples from OpenImages to avoid redundancy with the labeled portion of Union14M.

Diverse Text Styles

As depicted in the above image, Union14M includes a wide variety of text styles. These styles range from curved, slanted, and vertical text to challenging scenarios with blur, complex backgrounds, and occlusion. The dataset also captures text from diverse real-world environments, such as street signs and brand logos.

One important aspect is the large presence of vertical text, which is quite common in real-world scenarios but often underrepresented in synthetic datasets.

Rich Vocabulary

The vocabulary used in synthetic datasets is often derived from common corpora, but real-world text can vary significantly, including combinations not typically found in standard datasets—such as license plate numbers or mixed Chinese-English pinyin. As shown above, the vocabulary size of Union14M-L is nearly double that of synthetic datasets, providing a richer and more representative collection of real-world scenarios.

This expanded vocabulary enhances the depth and breadth of analysis available for STR models.

Model Architecture

After constructing the Union14M dataset, the authors proposed a self-supervised learning-based solution: MAERec.

This model takes advantage of self-supervised pretraining to leverage the 10 million unlabeled images from Union14M-U.

The core of MAERec is based on Vision Transformers (ViT), which excel at masked image modeling.

The input image is divided into 4 × 4 image patches and passed through the ViT backbone network. The output sequence is then processed by an autoregressive decoder (using the Transformer decoder from SATRN) to generate the final text prediction.

For the model comparisons, only the number of character classes was standardized to 91 classes (digits, uppercase and lowercase letters, symbols, and spaces); the other hyperparameters followed each model's original setup.

Pretraining and Fine-Tuning

To fully utilize the large number of unlabeled images from Union14M-U, the authors adopted the MAE framework for pretraining, with some modifications.

The examples show recognizable text reconstructions even with a 75% masking ratio. This suggests that pretraining captures useful visual structure. The downstream recognition experiments evaluate whether the learned features transfer.

After pretraining, the pretrained ViT weights are used to initialize MAERec, followed by fine-tuning on Union14M-L.

Discussion

The Harsh Reality

The authors evaluated 13 representative STR models, all trained on synthetic datasets, and tested them on Union14M-L. As shown in the table above, these models suffered a significant performance drop on Union14M-L, with the average accuracy falling by:

  • 20.50 percentage points!

Uncovering the Challenges

To identify the types of errors common to these 13 models, the authors assigned a difficulty score to each sample in Union14M-L, focusing on those that most models failed to recognize correctly. From these, they identified four unresolved challenges and three additional real-world scenarios that have been underexplored in previous research.

  • Curved Text

    As shown in image (a), recognizing curved text has been a focus in recent years, with two main approaches:

    1. Rectification-based models
    2. Models leveraging 2D attention mechanisms

    While these methods have performed well on datasets like CUTE, the proportion of curved text in these datasets is small, and the curvature tends to be moderate. When faced with highly curved text, existing models still struggle.


  • Multi-Oriented Text

    In image (b), text appears in vertical, slanted, or mirrored orientations on various surfaces, such as vertical text on signboards or slanted text due to camera angles. Most STR models assume near-horizontal text alignment, neglecting multi-oriented text challenges.

    These models often resize images to a fixed height (e.g., 32 pixels) while maintaining the aspect ratio. However, such resizing can compress vertical or slanted text, making it harder to recognize.


  • Artistic Text

    Image (e) shows artistic text with unusual fonts, effects, layouts, or complex backgrounds. The challenge is generalizing to styles rarely seen during training. An unseen style does not automatically turn this into a formally defined zero-shot or few-shot class-recognition task, nor does it establish that a specialized network is required. Limited coverage of these styles in synthetic training data remains an important concern.


  • Contextless Text

    As shown in image (f), contextless text refers to text that carries no inherent meaning or is not found in dictionaries, such as abbreviations or random combinations of letters, numbers, and symbols. Even with clear backgrounds and minimal distortion, models may misinterpret these texts due to over-reliance on semantic knowledge from training corpora.

    For example, a model might mistakenly recognize “YQJ” as “you,” which could be dangerous in applications like license plate recognition, invoice processing, or ID verification, where misrecognition can lead to security risks or financial losses.


  • Salient Text

    Image (c) depicts salient text, where irrelevant characters coexist with the primary text. When multiple texts of different sizes are adjacent or overlapping, these distractors may be included in the recognition, reducing model accuracy.

    In the detection phase, ROI masking strategies (as proposed by Liao et al.) can help remove such distracting characters. However, when detection models fail to locate text regions accurately, it becomes crucial for recognition models to quickly identify key regions for robust performance.


  • Multi-Words Text

    As seen in image (d), multi-word text appears in contexts like logos or phrases, where a single word is insufficient to convey complete meaning. Most STR models are trained on synthetic datasets with individual word-level annotations, making them prone to ignoring spaces between words.

    The authors observed that models often concatenate multiple words into one or modify some visible characters to match grammatical rules. For instance, "Live to Evolve" might be incorrectly recognized as "liveroee", as the model interprets it as a single word.


  • Incomplete Text

    As shown in image (g), text can become incomplete due to occlusion or inaccurate detection bounding boxes. When the beginning or end of a word is cropped, models might automatically fill in the missing parts to produce a complete prediction, even though the missing characters are not visible.

    This behavior is especially prominent in systems relying on language models, where predictions are heavily influenced by prior linguistic knowledge. However, this auto-completion feature can reduce reliability in certain applications. For example, if an image only shows “ight,” the model might complete it as "might" or "light", but the ideal output would be just "ight," leaving the decision to the downstream system for further anomaly detection.

    Therefore, it is essential to thoroughly evaluate the auto-completion feature and carefully assess its potential impact on downstream applications.

The Benchmark Dataset

To thoroughly evaluate the performance of STR models in real-world scenarios and facilitate further research on the seven key challenges, the authors developed a challenge-oriented benchmark dataset:

  • Union14M-Benchmark

This benchmark has eight subsets: 400,000 general samples and 9,383 challenge-specific samples, for a total of 409,383 images.

Experimental Results

To comprehensively compare the models, the authors first reported results using models trained on synthetic datasets (MJ + ST):

Next, they re-trained the models using Union14M-L:


Compared to traditional benchmark datasets, models trained on synthetic data showed a significant 48.5-percentage-point drop in average accuracy on the Union14M-Benchmark. However, when models were trained on Union14M-L, the accuracy drop was mitigated to 33.0 percentage points, indicating that real-world text images are far more complex than those in the six common benchmarks.

Moreover, models trained on Union14M-L achieved accuracy improvements of 3.9 percentage points on standard benchmarks and 19.6 percentage points on the Union14M-Benchmark. These results show limitations of the synthetic training sets used in this experiment and highlights the generalization benefits of training with real-world datasets.

When the 13 comparison models were trained only on Union14M-L, the highest average accuracy on the Union14M-Benchmark was 74.6%, excluding the incomplete-text subset. This reinforces that STR remains unsolved. While large-scale real-world data provides some performance gains, more research is needed to overcome these challenges.

On the incomplete-text subset, all models trained with synthetic data lost accuracy after the first or last character was cropped. The average drops in the WAICS metric were:

  • Language models: 10.2 percentage points drop
  • CTC models: 5.6 percentage points drop
  • Attention models: 5.9 percentage points drop

The authors attribute this drop to the error-correction behavior of language models, where the models attempt to auto-complete text perceived as missing characters. This issue was somewhat alleviated when models were trained on Union14M-L, possibly due to the larger vocabulary, which reduced overfitting to training corpora. However, this auto-completion problem persists and requires further investigation.

Conclusion

This paper presents a wealth of insights, addressing both current model limitations and dataset challenges. The creation of Union14M was a significant undertaking, providing a more comprehensive real-world text recognition dataset that allows for more thorough model evaluation.