Back to paper library
arXiv 2023Text recognitionPaper notes

OCR Scaling Law

Scaling Law for OCR

Original paper

An Empirical Study of Scaling Law for OCR

Paper authors

Miao Rang · Zhenni Bi · Chuanjian Liu · Yunhe Wang · Kai Han

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:2401.00028
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionocrscalinglaw2023,
  title = {{An Empirical Study of Scaling Law for OCR}},
  author = {Miao Rang and Zhenni Bi and Chuanjian Liu and Yunhe Wang and Kai Han},
  year = {2023},
  eprint = {2401.00028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.00028}
}
On this page

A few years ago, researchers in the NLP field attempted to summarize the scaling law for models, compiling several simple formulas.

Now, it’s already 2024, yet no paper seems to have addressed this issue in the OCR field.

To address this gap, Huawei's Noah's Ark Lab took on the challenge.

Defining the Problem

First, we need to know whether bigger models really help before investing more resources. After all, who would want to fund a larger model if the investment brings no benefit?

In the NLP domain, this question was explored years ago, with subsequent research delving even deeper. Examples include:

These studies were focused on NLP, but researchers in OCR seem to have paid little attention to this question.

Since people are hesitant to invest resources without understanding the risks, this area clearly needs research to bridge the gap.

Solving the Problem

This paper does not propose a new model architecture. Instead, the authors design an experimental framework to examine the scaling law of models from the following perspectives:

  1. Model Size: Refers to the number of parameters in the model, ranging from 50 million to 1 billion.
  2. Dataset Size: Refers to the size of the training dataset, ranging from 1 million to 1 billion samples.
  3. Computation Time: Refers to the time required to train the model, ranging from 100 hours to 1,000 hours.

These are the ranges outlined in the paper's introduction; the models and data volumes used in each experiment are given in the corresponding tables below.

In addition to this, the authors introduce a new dataset: REBU-Syn. This dataset, proposed in the paper, combines real and synthetic data, curated from existing publicly available datasets.

Model Scaling

The first aspect to explore is model size.

The authors reference two notable previous studies: TrOCR and PARSeq.

TrOCR is a text recognition model based entirely on the Transformer architecture. It was the first to combine vision Transformers with text Transformers for OCR tasks.

To analyze how model size impacts performance, the authors scale TrOCR models from TrOCR-S to TrOCR-H, with parameter sizes ranging from 43.09 million to 1.037 billion.

  • TrOCR-S: Uses DeiTSMALL_\text{SMALL} as the encoder and MiniLM as the decoder.
  • TrOCR-B: Uses BEITBASE_\text{BASE} as the encoder and RoBERTaLARGE_\text{LARGE} as the decoder.
  • TrOCR-L and TrOCR-H: Use BEITLARGE_\text{LARGE} as the encoder and RoBERTaLARGE_\text{LARGE} as the decoder.

PARSeq uses a Transformer-based encoder-decoder architecture.

  • The Encoder employs a ViT model to extract image features. It removes the [CLS] token and feeds all output tokens to the decoder.
  • The Decoder adopts a Transformer decoder similar to preLayerNorm, with the number of attention heads defined as nhead=dmodel32nhead = \frac{d_{model}}{32}.

The authors scale PARSeq models to four sizes to study the scaling laws for large OCR models:

  • PARSeq-S: The base version.
  • PARSeq-B, PARSeq-L, and PARSeq-H: Gradually expanded model sizes, ranging from 22.51 million to 682 million parameters. Detailed configurations are shown in the table below.

Data Scaling

In the early stages, model training primarily relied on synthetic data. However, as more real-world data became available, training models on real data proved to be more efficient. To scale up the training dataset, the authors created a new dataset: REBU-Syn, which consists of approximately 6 million real-world samples and 18 million publicly available synthetic samples, in a synthetic:real ratio of 3:1.

The real-world dataset is divided into the following components:

  • R Group: Includes common real-world sources such as COCO-Text, RCTW17, UberText, ArT, LSVT, MLT19, ReCTS, TextOCR, and OpenVINO.
  • E Group: Adds two real datasets commonly used in text detection, Total Text and CTW1500, as another source of real-world samples.
  • B Group: Contains training data from benchmark datasets like IIIT 5k-word, Street View Text, ICDAR13, and ICDAR15.
  • U Group: Draws from Union14M-L, which contains about 4 million labeled images across 14 datasets; this paper uses about 3.1 million after its compilation.

The synthetic dataset is composed of the following:

  • Commonly Used Datasets:

    • MJSynth (MJ): 8.9 million samples.
    • SynthText (ST): 5.5 million samples.
  • Additional Synthetic Data:

    • Curved SynthText (CST): Designed for detecting curved text.
    • SyntheAdd (SA): Generates data containing rare characters, such as punctuation marks.

In addition to public synthetic datasets, the authors generated more synthetic data using two tools: TextRecognitionDataGenerator and SynthText.

The text corpus was drawn from 700,000 entries from widely used English-language corpora globally. Backgrounds for the synthetic data were selected from natural scene images provided by SynthText.

Experimental Setup

The authors defined different hyperparameter configurations for each model architecture:

  • TrOCR Models
    • TrOCR-S: Batch size 10241024, learning rate 4×10−44 \times 10^{-4}.
    • TrOCR-B: Batch size 256256, learning rate 1×10−41 \times 10^{-4}.
    • TrOCR-L: Batch size 128128, learning rate 4×10−54 \times 10^{-5}.
  • PARSeq Models
    • A consistent learning rate of 7×10−47 \times 10^{-4} was used, with batch sizes as close to 10241024 as possible.

For text tokenization, BPE (Byte Pair Encoding) [68] and SentencePiece [37] were used to split text lines into subword units:

The primary evaluation metric was word accuracy. To ensure fair comparisons across models and datasets, the final output strings were standardized to a common set of 3636 characters (lowercase letters and digits).

Discussion

Here EE is the average word error rate across six test sets, so lower is better. The authors study its relationship with three variables:

  • Number of Model Parameters (NN): Reflecting the complexity and capacity of the model.
  • Amount of Training Data (DD): Representing the scale of the data available for training.
  • Model Computation Cost (CC): Represented in these experiments by training time; Figure 2(c) labels the horizontal axis as training hours on eight GPUs.

The goal of the experiments is to describe and validate the power-law relationship between recognition error EE and the variables NN, DD, and CC. The study also aims to determine if a smooth and scalable trend exists, enabling performance to improve effectively with increased resource allocation.

Power-Law for Model Size

  • TrOCR Models

    The number of parameters NN and recognition error EE exhibit a power-law relationship, expressed as:

    E(N)=(1.97×104N)0.223E(N) = \left(\frac{1.97 \times 10^4}{N}\right)^{0.223}

    The power-law formula is fitted based on the first three models (TrOCR-S, TrOCR-B, and TrOCR-L). The final model, TrOCR-H, aligns closely with the fitted curve, supporting the fit for these models and data conditions, without establishing unlimited extrapolation.

  • PARSeq Models

    Similar to TrOCR, the number of parameters NN and error EE in PARSeq follow a power-law relationship:

    E(N)=(6.316×10−74N)0.018E(N) = \left(\frac{6.316 \times 10^{-74}}{N}\right)^{0.018}

Power-Law for Dataset Size

  • TrOCR Models:

    Table 6 compares TrOCR-B word accuracy with 0.75 million to 75 million training samples. The paper prints the following fit for error EE:

    E(D)=(1.84×105D)−0.3271E(D) = \left(\frac{1.84 \times 10^5}{D}\right)^{-0.3271}

    Average accuracy in the table rises from 50.35% to 93.09%. Yet the printed negative exponent makes EE rise as DD increases, opposite to the falling error curve in Figure 2(b) and the table. We cannot use Equation (3) as printed to predict performance.

  • PARSeq Models:

    As R, B, E, U, and synthetic data are added in Table 7, average accuracy rises from 95.57% to 96.85%. Data sources and volume change together, so this table alone cannot isolate the effect of volume.

Power-Law for Computational Resources

Figure 2(c) plots average word error against training time. The paper prints Equation (4) as:

E(C)=(4.45×104C)−0.3271E(C) = \left(\frac{4.45 \times 10^4}{C}\right)^{-0.3271}

The grey fitted line falls as training time increases, but Equation (4)'s negative exponent implies the opposite. The plot legend also shows −0.764-0.764, while Equation (4) prints −0.3271-0.3271. We can read the observed trend from the plot but cannot treat the printed equation as a reliable prediction formula.

Ratio of Synthetic to Real Data

Real data generally outperforms synthetic data in training efficiency. However, due to its high cost and limited availability, synthetic data remains indispensable.

The authors fixed the amount of real data and increased synthetic data to 0.5–5 times that amount.

Table 9 reports the best accuracy among the tested ratios, 96.85%, at real:synthetic = 1:3. Accuracy declines at four and five times the real-data amount. The authors suggest a shift toward the synthetic distribution as a possible explanation.

Necessity of Pretraining

The effectiveness of pretrained models in improving OCR performance has been a topic of debate, with inconsistent findings in previous studies.

In this paper, the authors compared models pretrained on ImageNet-21k with models trained from scratch for OCR tasks.

In Table 10, PARSeq-L with ImageNet-21k pretraining averages 96.74%, below the 97.03% scratch-trained result on the same training data. The authors report a similar observation for CLIP4STR, but this table gives no CLIP4STR figures. These results suggest that non-OCR pretraining does not necessarily help recognition under these settings.

The OCR-related training sequence shows a further gain here. PARSeq-S trained on REBU-Syn averages 96.85%, above the 96.12% result using only real REBU data. Fine-tuning the REBU-Syn-trained model on REBU real data brings the Table 10 result to 96.96%; the paper's prose instead says 97.01%.

Based on these findings, the authors recommend the following best practices:

  1. Train the model on a comprehensive dataset containing both synthetic and real data.
  2. Fine-tune the model with real data to achieve further accuracy improvements.

Comparison with Other Models

The authors retrained CLIP4STR using a 3:1 ratio of synthetic to real data and fine-tuned from task-related pretrained models.

In Table 11, the originally reported CLIP4STR-B result is 96.89%, while the REBU-Syn-trained model reaches 97.25%, a difference of 0.36 percentage points. The rows use different training data, so we cannot attribute the difference solely to pretraining or the data ratio. CLIP4STR-L reaches a 97.42% sample-weighted average word accuracy across six common test benchmarks, the highest average in the table.

Conclusion

Within the OCR model families, data, and compute range studied, the observed error reductions can be approximated by power laws. These fits can inform resource planning, but a change in task, data quality, or model family requires fresh validation.

During training, the authors recommend using a 1:3 ratio of real to synthetic data for optimal performance. Additionally, they demonstrate that using task-specific pretrained models significantly enhances performance.