General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2409.01704
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textspottinggot2024,
title = {{General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model}},
author = {Haoran Wei and Chenglong Liu and Jinyue Chen and Jia Wang and Lingyu Kong and Yanming Xu and Zheng Ge and Liang Zhao and Jianjian Sun and Yuang Peng and Chunrui Han and Xiangyu Zhang},
year = {2024},
eprint = {2409.01704},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2409.01704}
}
This end-to-end OCR paper is a collaboration involving StepFun, Megvii, the University of Chinese Academy of Sciences, and Tsinghua University.
Problem Definition
OCR (Optical Character Recognition) is a widely used technology, with traditional systems typically relying on a multi-modular pipeline design, involving detection, cropping, and recognition modules. Each module has its specific function, and models are often developed for specific tasks, which can result in suboptimal overall performance and complicated maintenance.
The authors use “OCR 1.0” and “OCR 2.0” to frame their approach; these are not universal industry standards. Their unified model pursues the following goals:
- It is a unified end-to-end model.
- It addresses the bottlenecks faced by traditional and LVLM models in OCR tasks.
- It can handle various generalized OCR tasks.
This model is named GOT-OCR2.0!
Problem Solving
Model Architecture
The figure above not only illustrates the model architecture but also the training process.
The model architecture consists of three components:
- Vision Encoder: This is the image encoding layer responsible for feature extraction from images.
- Linear: This is the linear transformation layer that bridges the encoder and decoder feature dimensions.
- Text Decoder: This is the text decoding layer, which takes the image information and generates text.
The training process is divided into three phases:
Phase 1 Training
The authors used the ViTDet architecture as the encoder, benefiting from its local attention mechanism that significantly reduces the computational cost for high-resolution images. The encoder has around 80 million parameters, and the model was trained using 5 million image-text pairs.
The encoder's last two layers follow the Vary setup, transforming a 1024×1024×3 input image into 256×1024 image tokens. These tokens are projected through a 1024×768 linear layer to match the dimension of the language model (OPT-125M).
During preprocessing, images of varying shapes are resized directly to a 1024×1024 square. This standardizes the input but can distort the original aspect ratio.
Training Data
The training data is categorized into two types:
-
Natural Scene Data
English images were sampled from the Laion-2B dataset, while Chinese images were taken from the Wukong dataset. The text in these scenes was pseudo-labeled using the PaddleOCR tool. In this step, the authors collected 2 million data samples, evenly split between English and Chinese.
The labels are processed in two ways: first, the bounding boxes are removed and the text is combined from top to bottom and left to right; second, the original images are cropped around the text boxes, yielding another 1 million scene-text image–text pairs.
-
Document-level Data
The authors collected open-source PDF-style files from Common Crawl and extracted dense text content using PyMuPDF (imported as
fitz). They acquired 1.2 million full-page image–text pairs and 800,000 line- and paragraph-level image fragments by cropping PDF images using parsed bounding boxes.
Phase 2 Training
After pretraining the vision encoder, the model's knowledge was expanded through "multi-task joint training."
At this stage, the authors replaced the OPT-125M model used in Phase 1 with the Qwen-0.5B model, which has 500 million parameters and contains multilingual priors. The linear embedding layer's dimensions were adjusted to 1024×1024 to match Qwen-0.5B's input channels. GOT adopts an encoder-decoder paradigm, with a total parameter count of approximately 580 million.
The input image for GOT is 1024×1024 pixels, compressed into 256 tokens. The decoder references these tokens to predict OCR results, with a maximum length of up to 8K.
In this phase, the authors aimed to inject more OCR knowledge by exploring various data generation methods and engines:
-
Pure OCR Data
80% of the data prepared for Phase 1 was used as pure OCR data, and a handwritten text recognition subtask was added, covering multiple languages and handwriting styles. Datasets included Chinese CASIA-HWDB2, English IAM, and Norwegian NorHand-v3.
-
Mathpix-markdown Format Data
To maintain high readability of outputs, especially for mathematical formulas and tables, the authors used various methods to collect a large amount of formatted data:
- Math Formulas: Extracted .tex source files from Arxiv, obtaining around 1 million formula segments, which were converted to Mathpix format and rendered into PNG images.
- Molecular Structures: Extracted 2 million SMILES strings from ChEMBL_25 and generated about 1 million molecular-structure image/text pairs using Mathpix and rdkit.Chem tools.
- Tables: Extracted 300,000 table sources from .tex files and rendered them into images using LATEX.
- Full-page Data: Obtained 500,000 pairs of English Markdown PDF-text data and 500,000 pairs of Chinese Markdown data using the Nougat method. Additionally, the authors directly labeled 200,000 internal data points, including books, papers, and financial reports, using Mathpix.
-
Broader OCR Data
To enable GOT to handle more general optical character recognition tasks, the authors collected data for three related challenge tasks:
- Sheet Music: Rendered approximately 500,000 single-system sheet music data points using the GrandStaff dataset.
- Geometric Figures: Constructed basic geometric figures (e.g., circles, rectangles, triangles, simple function curves) using TikZ, resulting in about 1 million geometric TikZ data points.
- Charts: Referenced OneChart, rendering chart data using Matplotlib and Pyecharts, generating 2 million chart data points, with half from Matplotlib and the other half from Pyecharts.
Phase 3 Training
After Phase 2, the model can already handle various OCR tasks. In the final phase, the authors freeze the vision encoder and post-train the decoder to add three new features:
-
Fine-grained Interactive OCR Data Engine
Fine-grained OCR is a highly interactive feature that allows for region-level visual perception controlled by spatial coordinates or color. Users can specify regions of interest (RoI) by adding bounding boxes (box-guided OCR) or color-coded text (color-guided OCR) in the query prompt, avoiding unnecessary character output.
Natural scene fine-grained OCR data comes from open datasets such as RCTW, ReCTS, ShopSign, and COCO-Text. Their text boxes are used to generate region- and color-prompted OCR samples. For documents, the authors filter out scanned PDFs and parse the others with PyMuPDF/PDFminer, recording page images, line or paragraph boxes, and their text as box-guided training labels.
-
Large Image OCR with Multi-cropping Data Engine
GOT supports a 1024×1024 input resolution, enough for many scene OCR and A4 PDF tasks. For larger images, such as two PDF pages stitched side by side, it uses 1024×1024 windows with at most 12 crops. The authors also stitch PDF pages horizontally or vertically to synthesize about 500,000 image–text pairs for this feature.
-
Batch OCR for Multi-page PDF Files
The authors bundle multiple PDF pages into one OCR request so the model can handle content that is difficult to split cleanly at page boundaries, rather than processing each page with a separate for loop. The context and output length still limit the pages it can handle together.
For training, they randomly select 2–8 pages from Mathpix-format PDF data and combine them into one OCR task. Each page has fewer than 650 text tokens so that the overall sequence stays within 8K. This produces about 200,000 multi-page OCR samples, mostly interleaving Chinese and English pages.
Training Strategy
The authors used 64 L40S GPUs (8 × 8) for training, adopting a three-phase training approach:
- Pretraining Phase: The entire model was trained using the AdamW optimizer, with a global batch size of 128, over 3 epochs. The initial learning rate was 1e-4, and the maximum token length was set to 4096.
- Joint Training Phase: The maximum token length was increased to 6000, using the same optimizer setup as in pretraining, and trained for 1 epoch.
- Post-training Phase: The maximum token length was extended to 8192, with an initial learning rate of 2e-5, and trained for 1 epoch to support multi-block/page OCR functionality.
Discussion
Plain Text Document OCR Performance
The table reports edit distance, F1, precision, recall, BLEU, and METEOR using word-level segmentation. On the Fox document benchmark, GOT (580M) reaches F1 scores of 0.972 in English and 0.980 in Chinese; its English recall and METEOR are slightly below Qwen-VL-Max.
Scene Text OCR Performance
The authors collected 400 natural images, half Chinese and half English, and manually corrected the annotations for this scene-text OCR benchmark. Because the text in these images is relatively short, they calculate the metrics with character-level segmentation.
GOT scores 0.926 English F1 and 0.928 Chinese F1 on these images. Its precision in both languages is below InternVL-ChatV1.5, so this does not show that it leads every scene-text metric.
Formatted Document OCR Performance
The authors use 90 sample pages, labeled with Mathpix and manually corrected. On this set, switching from a single 1024×1024 input to multiple crops raises overall F1 from 0.942 to 0.953, formula F1 from 0.749 to 0.865, and table F1 from 0.867 to 0.878.
Fine-grained OCR Performance
In the table's English and Chinese box-guided and color-guided document OCR tests, GOT's F1 exceeds Fox's in each setting. For example, Chinese color-guided F1 rises from Fox's 0.884 to GOT's 0.957.
Conclusion
GOT is relatively simple in structure and focuses specifically on pure OCR tasks, demonstrating excellent performance across multiple tasks. Its ability to integrate various OCR tasks, including document-level text recognition, scene text recognition, fine-grained interactive OCR, formatted document recognition, and more general character recognition, makes it a highly flexible model with future potential.
The model is available on Hugging Face: GOT-OCR2.0
You can test its capabilities on the online demo: Online Demo