Character Region Awareness for Text Detection
- Publication
- CVPR 2019Conference
- First submitted to arXiv
- arXiv:1904.01941
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textdetectioncraft2019,
title = {{Character Region Awareness for Text Detection}},
author = {Youngmin Baek and Bado Lee and Dongyoon Han and Sangdoo Yun and Hwalsuk Lee},
year = {2019},
eprint = {1904.01941},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1904.01941}
}
Today, let's explore a well-known text detection paper, CRAFT.
Unlike most text detection methods, CRAFT focuses on character-level text detection.
Problem Definition
Character-level text detection is not a common approach, primarily due to the lack of available data.
A previous paper, Wordsup, attempted to train a character-level text detector using a weakly supervised framework, but it relied on rectangular anchors to locate characters.
The authors of CRAFT believed this method to be insufficiently precise, so they proposed an improved approach based on this inspiration.
Solution Approach
Model Architecture
The model architecture follows a conventional Backbone + Neck structure.
As shown above, we have labeled the diagram: the Backbone, marked in red, uses VGG16-BN. The upward fusion branch, marked in blue, uses U-Net-like skip connections and concatenates features from different scales.
One unusual aspect of the diagram is that the Backbone extends to Stage 6. The paper does not say this stage was added to handle especially large text. Its character-level design instead focuses on cues within a character and between adjacent characters.
Lastly, the prediction head uses the highest-resolution fused feature map, about half the input image size in each dimension, to predict two score-map channels: Region Scores and Affinity Scores.
But what are Region Scores and Affinity Scores? Let’s dive into that next.
Label Generation
Before discussing the prediction head, we need to explain how the labels are generated.
The authors first use synthetic SynthText data with character-box annotations. Real images usually have only word boxes and transcriptions, so character boxes are generated later through weak supervision.
For each training image, the character-level boundary boxes are used to generate:
- Region Score: a smooth target score that is high near a character’s center.
- Affinity Score: a smooth target score for the connection between adjacent characters.
Instead of binary segmentation masks, the authors encode these scores using Gaussian heatmaps to represent character-centered target scores; these are not calibrated probabilities. The process of calculating these Gaussian values is complex, especially for characters under perspective distortions. To improve efficiency, the following steps are used to generate the ground-truth labels:
- Prepare a 2D isotropic Gaussian map.
- Compute a perspective transformation between the Gaussian map and each character box.
- Warp the Gaussian map to match the character box region.
For affinity scores, the diagonals of each character box divide it into upper and lower triangles. The centers of those triangles in a pair of adjacent character boxes serve as the corners of an affinity box.
Weak Supervision Learning
The authors address the issue of insufficient training data by employing a weakly supervised method to generate character boxes from word-level annotations.
Referring to the diagram above, the blue line shows the process for fully annotated training images. These are simply fed into the model for loss calculation.
For images without character annotations, the green and red lines come into play.
The green line represents training on real images with generated character pseudo-labels. The red line shows how those pseudo-labels are made: crop words from the real images, which have word boxes and transcriptions but no character boxes, and let the current model predict their region scores.
Like this:
Then, the watershed algorithm is applied to segment the region score into character areas, generating character-level bounding boxes. These coordinates are then mapped back to the original image to create character-level labels, as shown below:
Once the labels are obtained, the standard loss calculation process resumes.
For each word-level training sample , the confidence score is defined as:
Here is the annotated character count and the count of predicted character components. The score measures agreement in count, not the correctness of their positions. is the annotated word region.
The pixel-wise confidence map is defined as:
The training loss function is defined as:
Here and are the supervision maps for region and affinity scores: ground-truth labels for synthetic images and pseudo-labels for real images. and are the model predictions.
As the training progresses, the model becomes better at predicting characters, and the confidence score increases. Early in training, region scores for unfamiliar text in natural images tend to be lower.
What happens when the confidence score is below 0.5?
The authors set a rule: ignore the predicted character boxes in this case, as they would negatively impact training!
Instead, the authors assume a constant character width, divide the word region evenly according to the number of transcribed characters, and set that word's confidence score to 0.5. Although this yields imprecise character boxes, it helps the model learn previously unseen text appearances.
Word-Level Bounding Boxes
The final output during inference can be in various forms, including word boxes, character boxes, or polygons. For datasets like ICDAR, the evaluation is based on word-level Intersection-over-Union (IoU).
Here’s a simplified description of how word-level bounding boxes (QuadBoxes) are generated based on the predicted region and affinity scores.
First, initialize a binary image , which covers the entire image, with all pixels set to 0.
If
or
then set to 1, where is the region threshold and is the affinity threshold.
Second, perform connected component labeling (CCL) on . Finally, using the minimum bounding rectangle that encloses each connected component, the QuadBox is formed. This can be achieved using OpenCV functions like connectedComponents and minAreaRect.
Curved Text Polygons
To handle curved text effectively, polygons can be generated to enclose entire character regions. The steps for generating these polygons are as follows:
- Identify the local maxima along the scan direction of the character region and connect them to form the yellow central lines shown in the diagram.
- To prevent uneven polygon results, set all local maxima lines to an equal maximum length.
- Rotate the local maxima lines to be perpendicular to the centerline, reflecting the character's slant (indicated by the red arrows). These local maxima endpoints become candidate control points for the polygon.
- Finally, extend the two outermost slanted local maxima lines outward along the centerline to fully enclose the text area, forming the final control points (green points).
Training Datasets
-
SynthText
This dataset is used for pre-training the model. It is a large-scale dataset containing about 800,000 synthetic images. These images consist of natural scenes blended with text in random fonts, sizes, colors, and orientations, giving them a realistic appearance.
-
CTW1500
CTW1500 is a challenging dataset for detecting long curved texts, constructed by Yuliang et al. It consists of 1000 training images and 500 testing images. Unlike traditional datasets (such as ICDAR 2015, ICDAR 2017 MLT), CTW1500 uses 14-point polygons to annotate text instances, which can describe arbitrary-shaped curved text.
-
Total-Text
Total-Text is a newly released curved text detection dataset, containing horizontal, multi-directional, and curved text instances. This benchmark consists of 1255 training images and 300 test images.
-
ICDAR 2013
ICDAR 2013 was released during the reading competition, used for focused scene text detection. It contains high-resolution images, with 229 for training and 233 for testing, including English text. The annotations are at the word level using rectangular boxes.
-
ICDAR 2015
This is a widely used dataset for text detection, consisting of 1500 images, with 1000 used for training and the rest for testing. The text regions are annotated as quadrilaterals.
-
MSRA-TD500
This dataset contains multilingual, arbitrarily oriented, long text lines. It includes 300 training images and 200 testing images, annotated at the line level. CRAFT was not fine-tuned on the MSRA-TD500 training set.
-
ICDAR 2017 MLT
IC17-MLT is a large-scale multi-lingual text dataset containing 7200 training images, 1800 validation images, and 9000 testing images. The dataset consists of scene images in 9 different languages.
Training Strategy
- Initial training is conducted on the SynthText dataset, for 50,000 iterations.
- The model is fine-tuned on ICDAR datasets with the word boxes and transcriptions needed for weak supervision. For regions labeled "DO NOT CARE" in IC15 and IC17, the confidence score is set to 0 to ignore them during training.
- All training is performed using the ADAM optimizer.
- During fine-tuning, the SynthText dataset is mixed at a 1:5 ratio to ensure proper separation of character regions.
- Online Hard Negative Mining uses a positive:negative ratio of 1:3.
- Basic data augmentation techniques, such as cropping, rotation, and color variations, are also applied.
The weakly supervised learning component requires two types of data: quadrilateral annotations and word length.
Datasets that meet these conditions include IC13, IC15, and IC17.
Other datasets, like MSRA-TD500, Total-Text, and CTW-1500, do not meet these requirements:
- MSRA-TD500 does not provide transcripts.
- Total-Text and CTW-1500 only provide polygon annotations.
After SynthText pretraining, CRAFT was fine-tuned on ICDAR datasets and tested on other datasets without additional fine-tuning.
Two models were trained by the authors. The first model was fine-tuned on IC15 and used to evaluate the IC15 dataset. The second model was jointly fine-tuned on IC13 and IC17 and used to evaluate the remaining five datasets. Fine-tuning ran for 25,000 iterations; aside from the stated SynthText mixture, no other natural images were added.
Discussion
Comparison with Other Methods
The table above lists h-mean scores of various methods on ICDAR and MSRA-TD500 datasets. To ensure fair comparison with end-to-end methods, the paper includes results for detection-only performance as reported in the original papers.
CRAFT has the highest h-mean in this table's four columns: 95.2 on IC13, 86.9 on IC15, 73.9 on IC17 and 82.9 on MSRA-TD500. Its IC13 speed is 8.6 FPS. The other methods' FPS figures come from their own papers and different test environments, so they are not directly comparable.
Another table reports results without fine-tuning CRAFT on the curved-text datasets: 83.6 h-mean on TotalText and 83.5 on CTW-1500. Both use polygons generated from character regions at inference. For the CTW-1500 result, however, the paper adds a LinkRefiner trained on CTW-1500 while keeping CRAFT frozen; this refines affinity scores for long text lines. That qualification matters when reading the 83.5 result.
For the MSRA-TD500 dataset, the annotations are line-based and include spaces between words. Therefore, a post-processing step is employed to merge word boxes: if the right side of one box is sufficiently close to the left side of another, the boxes are merged. Despite not fine-tuning on the TD500 training set, CRAFT outperformed all other methods on this dataset.
Visualization
In the top row, each input image is paired with its character region score map on the lower left and adjacent-character affinity score map on the lower right. The bottom row pairs each input with its region score map, showing how character centers follow curved words.
Conclusion
This paper highlights two key takeaways:
-
Weak Supervision Learning: This is an effective approach to address the issue of insufficient data, but it requires certain techniques to enhance model reliability. The character-count confidence weights pseudo-labels and reduces the influence of unreliable predictions. This is particularly useful for handling limited labeled data in real-world scenarios.
-
Effective Post-Processing: In text detection tasks, simple yet efficient post-processing steps can significantly enhance model performance. CRAFT’s post-processing, such as connected component labeling (CCL) and the use of rotating rectangles, reduces reliance on complex post-processing methods like non-maximum suppression (NMS) while effectively handling curved text in polygon generation.