Back to paper library
arXiv 2021AAAI 2023Text recognitionPaper notes

TrOCR

Pretraining is Justice

Original paper

TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

Paper authors

Minghao Li · Tengchao Lv · Jingye Chen …

Show all 9 authors

Minghao Li · Tengchao Lv · Jingye Chen · Lei Cui · Yijuan Lu · Dinei Florencio · Cha Zhang · Zhoujun Li · Furu Wei

Publication
AAAI 2023Conference
First submitted to arXiv
arXiv:2109.10282
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitiontrocr2021,
  title = {{TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models}},
  author = {Minghao Li and Tengchao Lv and Jingye Chen and Lei Cui and Yijuan Lu and Dinei Florencio and Cha Zhang and Zhoujun Li and Furu Wei},
  year = {2021},
  eprint = {2109.10282},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.10282}
}
On this page

This is a straightforward paper, ideal for developers who are new to this field and looking for a clear entry point.

Problem Definition

At the time, many OCR systems still used CNN backbones, adding Transformer self-attention to better understand text images. CTC handled sequence alignment in some recognition systems, and an external language model could improve their predictions; these played different roles.

Although these methods have seen some success, most models are still trained from scratch and do not fully utilize the advantages of large-scale pre-trained models in computer vision (CV) and natural language processing (NLP).

In this paper, the authors lead by example, showing how pre-trained models can be applied effectively to OCR tasks.

Solution

Model Architecture

The TrOCR model is composed of two parts: an image Transformer and a text Transformer.

Unlike traditional text recognition models, TrOCR doesn’t rely on a CNN backbone. Instead, it follows the approach used in ViT, resizing the input image to 384×384384 \times 384 and then splitting it into 16×1616 \times 16 patches before feeding it into the image encoder. Both the encoder and decoder use a standard Transformer architecture, with wordpiece units as the output for recognized text.

For the experiments, the image encoder is initialized from the ViT-style DeiT or BEiT models, and the decoder from RoBERTa or MiniLM. The decoder must use causal self-attention for OCR generation; its encoder-decoder cross-attention parameters are absent from those pretrained models and are initialized randomly.

Training Strategy

  • Phase One

    In the first phase of pre-training, the authors sampled two million pages from publicly available PDF files. These PDFs were converted into page images, and each text line was extracted to create a dataset of 684 million text line images. Since these PDFs are “digitally generated,” they offer high clarity in the text images.

  • Phase Two

    Using the open-source text generator TRDG and 5,427 handwritten fonts, the authors synthesized handwritten text line images with text content randomly drawn from Wikipedia pages. This stage generated a dataset of 17.9 million text lines, including the IIIT-HWS dataset.

    Additionally, around 53,000 real-world receipt images were collected. A commercial OCR engine recognized the text on these images, and each text line was cropped and normalized based on its coordinates.

    Finally, TRDG was used to generate around one million printed text line images with receipt fonts and other built-in fonts, resulting in a printed dataset with 3.3 million text lines.

    For scene text recognition, the second phase used the MJSynth and SynthText datasets, comprising about 16 million text images in total.

Parameter Settings

The image branch of TrOCR uses DeiT and BEiT models, while the text branch uses RoBERTa or MiniLM.

The DeiT model implementation and pre-trained weights come from the timm library, the BEiT and MiniLM models are from the official UniLM repository, and the RoBERTa model is sourced from Fairseq’s GitHub repository.

Training is conducted using 32 V100 GPUs for pre-training and 8 V100 GPUs for fine-tuning. All models use a batch size of 2,048 and a learning rate of 5×10−55 \times 10^{-5}.

The following details outline additional parameter settings:

  • Resolution and Patch Size: DeiT and BEiT encoders use an image resolution of 384×384384 \times 384, with a patch size of 16×1616 \times 16.
  • Encoder Architecture:
    • DeiTSMALL\text{DeiT}_{\text{SMALL}}: 12 layers, 384-dimensional hidden size, 6 attention heads.
    • DeiTBASE\text{DeiT}_{\text{BASE}} and BEiTBASE\text{BEiT}_{\text{BASE}}: 12 layers, 768-dimensional hidden size, 12 attention heads.
    • BEiTLARGE\text{BEiT}_{\text{LARGE}}: 24 layers, 1024-dimensional hidden size, 16 attention heads.
  • Decoder Architecture:
    • Small decoder: 6 layers, 256-dimensional hidden size, 8 attention heads.
    • Base decoder: 512-dimensional hidden size.
    • Large decoder: 12 layers, 1024-dimensional hidden size, 16 attention heads.

Discussion

Encoder-Decoder Combinations

Given the variety of encoder and decoder options available, which combination performs best?

The authors conducted experiments to evaluate different encoder-decoder pairings using the SROIE dataset. The results are summarized in the table below:

For encoders, the authors tested DeiT, BEiT, and ResNet-50; for decoders, they used RoBERTa.

In this architecture comparison trained only on SROIE, the BEiTBASE_{\text{BASE}} encoder with the RoBERTaLARGE_{\text{LARGE}} decoder has the highest F1 score, 79.36%. The separate ablation table tests the gains from pretrained initialization and the two pretraining stages.

Based on these findings, the authors selected three primary configurations for further experiments:

  • TrOCRSMALL_{\text{SMALL}}: Composed of the DeiTSMALL_{\text{SMALL}} encoder and MiniLM decoder (62M parameters).
  • TrOCRBASE_{\text{BASE}}: Composed of the BEiTBASE_{\text{BASE}} encoder and RoBERTaLARGE_{\text{LARGE}} decoder (334M parameters).
  • TrOCRLARGE_{\text{LARGE}}: Composed of the BEiTLARGE_{\text{LARGE}} encoder and RoBERTaLARGE_{\text{LARGE}} decoder (558M parameters).

Performance on the SROIE Dataset

The table above compares TrOCR models with other methods on SROIE. It evaluates text-line images cropped using ground-truth boxes, without measuring text detection.

In terms of language modeling, MSO Lab and CLOVA OCR models use LSTM layers, while H&H Lab uses GRU layers. In contrast, TrOCR utilizes a pure attention-based Transformer decoder.

TrOCRLARGE_{\text{LARGE}} reaches 96.58% F1 on this text-line recognition task, slightly above H&H Lab's 96.43%, without complex additional recognition post-processing.

Performance on the IAM Dataset

The table above compares the performance of TrOCR models to existing methods on the IAM handwriting dataset.

For Diaz et al.'s CTC model, adding an external language model (LM) lowers the character error rate (CER) from 3.53% to 2.75%. TrOCRLARGE_{\text{LARGE}} reaches 2.89% CER without an external LM. That is below Bluche and Messina's CTC result of 3.20%, but above Diaz et al.'s result with its extra internal data and LM; the training data also differ between methods.

Performance on STR

The performance of TrOCR was evaluated with fine-tuning on synthetic data alone, as well as fine-tuning on both synthetic and benchmark datasets (IC13, IC15, IIIT5K, SVT training sets). The results were then compared to existing mainstream and recent SOTA methods.

In the table's eight test columns, TrOCR fine-tuned on synthetic plus benchmark training data has the highest listed result on SVT, both IC13 subsets, SVTP, and CUTE. It does not lead on IIIT5K or either IC15 subset.

Conclusion

The key distinction of TrOCR from previous studies is its use of pre-trained models.

By making full use of pretrained models, the authors build a relatively simple Transformer encoder-decoder without a CNN backbone. The results show its potential across printed, handwritten, and scene-text recognition. The architecture is simpler to describe and maintain, although these results alone do not show that every TrOCR setting uses less computation than earlier models.