An Empirical Study of Scaling Law for OCR
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2401.00028
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textrecognitionocrscalinglaw2023,
title = {{An Empirical Study of Scaling Law for OCR}},
author = {Miao Rang and Zhenni Bi and Chuanjian Liu and Yunhe Wang and Kai Han},
year = {2023},
eprint = {2401.00028},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2401.00028}
}
A few years ago, researchers in the NLP field attempted to summarize the scaling law for models, compiling several simple formulas.
Now, it’s already 2024, yet no paper seems to have addressed this issue in the OCR field.
To address this gap, Huawei's Noah's Ark Lab took on the challenge.
Defining the Problem
First, we need to know whether bigger models really help before investing more resources. After all, who would want to fund a larger model if the investment brings no benefit?
In the NLP domain, this question was explored years ago, with subsequent research delving even deeper. Examples include:
These studies were focused on NLP, but researchers in OCR seem to have paid little attention to this question.
Since people are hesitant to invest resources without understanding the risks, this area clearly needs research to bridge the gap.
Solving the Problem
This paper does not propose a new model architecture. Instead, the authors design an experimental framework to examine the scaling law of models from the following perspectives:
- Model Size: Refers to the number of parameters in the model, ranging from 50 million to 1 billion.
- Dataset Size: Refers to the size of the training dataset, ranging from 1 million to 1 billion samples.
- Computation Time: Refers to the time required to train the model, ranging from 100 hours to 1,000 hours.
These are the ranges outlined in the paper's introduction; the models and data volumes used in each experiment are given in the corresponding tables below.
In addition to this, the authors introduce a new dataset: REBU-Syn. This dataset, proposed in the paper, combines real and synthetic data, curated from existing publicly available datasets.
Model Scaling
The first aspect to explore is model size.
The authors reference two notable previous studies: TrOCR and PARSeq.
TrOCR is a text recognition model based entirely on the Transformer architecture. It was the first to combine vision Transformers with text Transformers for OCR tasks.
To analyze how model size impacts performance, the authors scale TrOCR models from TrOCR-S to TrOCR-H, with parameter sizes ranging from 43.09 million to 1.037 billion.
- TrOCR-S: Uses DeiT as the encoder and MiniLM as the decoder.
- TrOCR-B: Uses BEIT as the encoder and RoBERTa as the decoder.
- TrOCR-L and TrOCR-H: Use BEIT as the encoder and RoBERTa as the decoder.
PARSeq uses a Transformer-based encoder-decoder architecture.
- The Encoder employs a ViT model to extract image features. It removes the
[CLS]token and feeds all output tokens to the decoder. - The Decoder adopts a Transformer decoder similar to preLayerNorm, with the number of attention heads defined as .
The authors scale PARSeq models to four sizes to study the scaling laws for large OCR models:
- PARSeq-S: The base version.
- PARSeq-B, PARSeq-L, and PARSeq-H: Gradually expanded model sizes, ranging from 22.51 million to 682 million parameters. Detailed configurations are shown in the table below.
Data Scaling
In the early stages, model training primarily relied on synthetic data. However, as more real-world data became available, training models on real data proved to be more efficient. To scale up the training dataset, the authors created a new dataset: REBU-Syn, which consists of approximately 6 million real-world samples and 18 million publicly available synthetic samples, in a synthetic:real ratio of 3:1.
The real-world dataset is divided into the following components:
- R Group: Includes common real-world sources such as COCO-Text, RCTW17, UberText, ArT, LSVT, MLT19, ReCTS, TextOCR, and OpenVINO.
- E Group: Adds two real datasets commonly used in text detection, Total Text and CTW1500, as another source of real-world samples.
- B Group: Contains training data from benchmark datasets like IIIT 5k-word, Street View Text, ICDAR13, and ICDAR15.
- U Group: Draws from Union14M-L, which contains about 4 million labeled images across 14 datasets; this paper uses about 3.1 million after its compilation.
The synthetic dataset is composed of the following:
-
Commonly Used Datasets:
- MJSynth (MJ): 8.9 million samples.
- SynthText (ST): 5.5 million samples.
-
Additional Synthetic Data:
- Curved SynthText (CST): Designed for detecting curved text.
- SyntheAdd (SA): Generates data containing rare characters, such as punctuation marks.
In addition to public synthetic datasets, the authors generated more synthetic data using two tools: TextRecognitionDataGenerator and SynthText.
The text corpus was drawn from 700,000 entries from widely used English-language corpora globally. Backgrounds for the synthetic data were selected from natural scene images provided by SynthText.
Experimental Setup
The authors defined different hyperparameter configurations for each model architecture:
- TrOCR Models
- TrOCR-S: Batch size , learning rate .
- TrOCR-B: Batch size , learning rate .
- TrOCR-L: Batch size , learning rate .
- PARSeq Models
- A consistent learning rate of was used, with batch sizes as close to as possible.
For text tokenization, BPE (Byte Pair Encoding) [68] and SentencePiece [37] were used to split text lines into subword units:
The primary evaluation metric was word accuracy. To ensure fair comparisons across models and datasets, the final output strings were standardized to a common set of characters (lowercase letters and digits).
Discussion
Here is the average word error rate across six test sets, so lower is better. The authors study its relationship with three variables:
- Number of Model Parameters (): Reflecting the complexity and capacity of the model.
- Amount of Training Data (): Representing the scale of the data available for training.
- Model Computation Cost (): Represented in these experiments by training time; Figure 2(c) labels the horizontal axis as training hours on eight GPUs.
The goal of the experiments is to describe and validate the power-law relationship between recognition error and the variables , , and . The study also aims to determine if a smooth and scalable trend exists, enabling performance to improve effectively with increased resource allocation.
Power-Law for Model Size
-
TrOCR Models
The number of parameters and recognition error exhibit a power-law relationship, expressed as:
The power-law formula is fitted based on the first three models (TrOCR-S, TrOCR-B, and TrOCR-L). The final model, TrOCR-H, aligns closely with the fitted curve, supporting the fit for these models and data conditions, without establishing unlimited extrapolation.
-
PARSeq Models
Similar to TrOCR, the number of parameters and error in PARSeq follow a power-law relationship:
Power-Law for Dataset Size
-
TrOCR Models:
Table 6 compares TrOCR-B word accuracy with 0.75 million to 75 million training samples. The paper prints the following fit for error :
Average accuracy in the table rises from 50.35% to 93.09%. Yet the printed negative exponent makes rise as increases, opposite to the falling error curve in Figure 2(b) and the table. We cannot use Equation (3) as printed to predict performance.
-
PARSeq Models:
As R, B, E, U, and synthetic data are added in Table 7, average accuracy rises from 95.57% to 96.85%. Data sources and volume change together, so this table alone cannot isolate the effect of volume.
Power-Law for Computational Resources
Figure 2(c) plots average word error against training time. The paper prints Equation (4) as:
The grey fitted line falls as training time increases, but Equation (4)'s negative exponent implies the opposite. The plot legend also shows , while Equation (4) prints . We can read the observed trend from the plot but cannot treat the printed equation as a reliable prediction formula.
Ratio of Synthetic to Real Data
Real data generally outperforms synthetic data in training efficiency. However, due to its high cost and limited availability, synthetic data remains indispensable.
The authors fixed the amount of real data and increased synthetic data to 0.5–5 times that amount.
Table 9 reports the best accuracy among the tested ratios, 96.85%, at real:synthetic = 1:3. Accuracy declines at four and five times the real-data amount. The authors suggest a shift toward the synthetic distribution as a possible explanation.
Necessity of Pretraining
The effectiveness of pretrained models in improving OCR performance has been a topic of debate, with inconsistent findings in previous studies.
In this paper, the authors compared models pretrained on ImageNet-21k with models trained from scratch for OCR tasks.
In Table 10, PARSeq-L with ImageNet-21k pretraining averages 96.74%, below the 97.03% scratch-trained result on the same training data. The authors report a similar observation for CLIP4STR, but this table gives no CLIP4STR figures. These results suggest that non-OCR pretraining does not necessarily help recognition under these settings.
The OCR-related training sequence shows a further gain here. PARSeq-S trained on REBU-Syn averages 96.85%, above the 96.12% result using only real REBU data. Fine-tuning the REBU-Syn-trained model on REBU real data brings the Table 10 result to 96.96%; the paper's prose instead says 97.01%.
Based on these findings, the authors recommend the following best practices:
- Train the model on a comprehensive dataset containing both synthetic and real data.
- Fine-tune the model with real data to achieve further accuracy improvements.
Comparison with Other Models
The authors retrained CLIP4STR using a 3:1 ratio of synthetic to real data and fine-tuned from task-related pretrained models.
In Table 11, the originally reported CLIP4STR-B result is 96.89%, while the REBU-Syn-trained model reaches 97.25%, a difference of 0.36 percentage points. The rows use different training data, so we cannot attribute the difference solely to pretraining or the data ratio. CLIP4STR-L reaches a 97.42% sample-weighted average word accuracy across six common test benchmarks, the highest average in the table.
Conclusion
Within the OCR model families, data, and compute range studied, the observed error reductions can be approximated by power laws. These fits can inform resource planning, but a change in task, data quality, or model family requires fresh validation.
During training, the authors recommend using a 1:3 ratio of real to synthetic data for optimal performance. Additionally, they demonstrate that using task-specific pretrained models significantly enhances performance.