TextFuseNet: Scene Text Detection with Richer Fused Features
- Publication
- IJCAI 2020Conference
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@inproceedings{textdetectiontextfusenet2020,
title = {{TextFuseNet: Scene Text Detection with Richer Fused Features}},
author = {Jian Ye and Zhe Chen and Juhua Liu and Bo Du},
year = {2020},
booktitle = {IJCAI},
url = {https://doi.org/10.24963/ijcai.2020/72},
doi = {10.24963/ijcai.2020/72}
}
We found the publication year, 2020, but could not pin down its month. The GitHub repository showed a September 2020 upload, so we filed the note under that month. If we found the wrong date, please tell us.
Problem Definition
The scene text detection problem has largely been framed as a segmentation task. Segmentation tasks are highly susceptible to interference from the background, especially in complex scenes where unexpected objects may occlude the text, significantly reducing the model's performance.
To improve model robustness, the authors of this paper also took a feature fusion approach. They argue that features should include:
- Character-level features: Shapes, sizes, and orientations of characters.
- Word-level features: Contextual and background information of the text.
- Image-level features: Overall characteristics of the entire image.
They proposed a novel model called TextFuseNet, which extracts these three types of features and fuses them together.
Solution
Model Architecture
The model might look a bit complicated, but let’s break it down into five parts:
- Backbone + Neck: A standard setup using ResNet-50 as the backbone network and FPN (Feature Pyramid Network) as the neck.
- RPN: The network responsible for generating candidate text regions, employing an RPN (Region Proposal Network) and RoIAlign.
- Head 1: Semantic segmentation branch, supplying global semantic features and segmentation results.
- Head 2: Detection branch, classifying words and characters and regressing their bounding boxes.
- Head 3: Mask branch, performing instance segmentation for words and characters.
The first part is quite familiar, so let’s focus on the second part onwards.
The Second Part: RPN + RoIAlign
The authors borrowed ideas from Mask R-CNN and Mask TextSpotter, using an RPN to generate text proposals for subsequent detection and segmentation branches. These proposals are refined via semantic segmentation and multi-layer feature extraction for final text detection.
The RPN predicts objectness scores and box offsets relative to predefined anchors on FPN feature maps. Applying those offsets produces candidate regions; the network does not generate the reference anchors themselves from scratch.
Once the RPN generates text region proposals, RoIAlign is used to extract different features, enabling simultaneous word and character detection.
The Third Part: Text Region Prediction Branch (Semantic Segmentation Branch)
This branch extracts global semantic features for text and background across the scene. When text blends into the background or has an irregular shape, that wider context can help detection. A 1×1 convolution aligns the channel counts of FPN features from different levels; the feature maps are then resized to the same size and fused.
The Fourth Part: Character-Level Detection Branch (Detection Branch)
This branch detects words and characters from the text proposals. At this stage it fuses word-level and global features; character-level features can enter only after characters have been detected. RoIAlign samples aligned proposal features without the quantization error of traditional RoIPooling. Classification and bounding-box regression then locate both words and characters.
The Fifth Part: Character-Level Segmentation Branch (Mask Branch)
The mask branch performs instance segmentation for detected words and characters. It predicts masks for their shapes, which helps with irregular text. Once the detection branch has found the instances, the mask branch fuses character-, word-, and global-level features along multiple paths to generate their masks.
Multi-Path Feature Fusion
The multi-path feature fusion architecture is the key innovation of this model, as it fuses character, word, and global features to improve both detection and segmentation accuracy.
-
Fusion in the Detection Branch:
In the detection branch, TextFuseNet extracts global and word-level features for each text proposal. Global features come from the semantic segmentation branch, while word-level features come from the FPN.
For each text proposal, RoIAlign samples global and word-level features into 7×7 outputs. Then, these features are fused using element-wise summation.
After fusion, the features are further processed through 3×3 and 1×1 convolution layers, used for classification and bounding box regression.
-
Fusion in the Mask Branch:
In the mask branch, for each word proposal, the character attributes are computed using the formula:
Here is word box , is character box , and . The fraction measures how much of the character box lies inside the word box; it is not IoU.
RoIAlign samples character features into 14×14 outputs, and these character-level features are fused via element-wise summation. The fused features are further processed by 3×3 and 1×1 convolutions to generate the final character features.
-
Three-Level Feature Fusion:
Finally, character-level, word-level, and global semantic features are fused. RoIAlign extracts corresponding features, which are combined and further refined through convolution layers. These fused and processed features generate richer, more distinctive representations for text instance segmentation.
Weak Supervision Learning
A common challenge for models involving character-level detection and segmentation is the lack of annotated data. Like CRAFT, the authors of TextFuseNet used weak supervision learning to address this issue. Readers interested in this method can refer to our previous introduction:
First, the authors pre-trained TextFuseNet on a dataset with complete character and word annotations, obtaining a pre-trained model capable of detecting both characters and words.
Next, they applied model to a dataset annotated only at the word level to detect potential character samples. For each image in dataset , the pre-trained model generated a set of character candidates , defined as follows:
Where represents the -th character candidate, is the predicted character class, is the confidence score, is the character bounding box, and is the character mask.
The authors keep candidates whose confidence score exceeds 0.1 and whose mask overlaps a word-level ground-truth region by more than 0.8 of the character-mask area. The accepted samples serve as character-level labels alongside the word-level annotations for fine-tuning.
Training Datasets
-
SynthText
A large-scale synthetic dataset containing about 800,000 images, used for pre-training the model. These images combine natural scenes with randomly generated text in various fonts, sizes, colors, and orientations, closely mimicking real-world scenarios.
-
CTW1500
A challenging dataset for curved text detection, consisting of 1,000 training images and 500 test images. Unlike traditional text datasets, CTW1500 includes text instances annotated with 14-point polygons, capable of describing arbitrarily curved text shapes.
-
Total-Text
A newly released dataset for curved text detection, comprising horizontal, multi-oriented, and curved text instances. The benchmark includes 1,255 training images and 300 test images.
-
ICDAR 2013
A dataset from the Reading Competition, consisting of 229 training images and 233 test images, all in high resolution. English text is annotated with rectangular boxes at the word level.
-
ICDAR 2015
A popular text detection dataset containing 1,500 images, with 1,000 for training and the rest for testing. Text regions are annotated using quadrilateral bounding boxes with four vertices.
Discussion
Ablation Study
The authors introduced two modules to improve text detection performance in TextFuseNet compared to the original Mask R-CNN:
- Multi-Level Feature Representation (MFR): Extracts character-, word-, and global-level features.
- Multi-Path Feature Aggregation (MFA): Fuses these three levels into richer features for text detection.
To validate the effectiveness of these two modules, the authors conducted an ablation study on the ICDAR 2015 and Total-Text datasets. Three different models were prepared for each dataset:
- Baseline: The original Mask R-CNN trained model.
- MFR: Mask R-CNN model with Multi-Level Feature Representation (MFR).
- MFR+MFA: The complete TextFuseNet model with both MFR and MFA.
ResNet-50 with FPN was used as the backbone network in these models.
As shown above, MFR alone improves both precision and recall. Against the baseline, F-measure increases by 2.8 percentage points on ICDAR 2015 and 2.7 points on Total-Text.
Adding MFA to MFR brings a further 1.8 and 1.6 percentage-point gain on ICDAR 2015 and Total-Text, respectively. Against the baseline, the complete model gains 4.6 and 4.3 points. These ablations support a contribution from both modules in the tested settings.
Comparison with Other Methods
As shown in the last two dataset groups, TextFuseNet's F-measure exceeds the earlier methods listed for CTW-1500 and Total-Text.
On CTW-1500, TextFuseNet with ResNet-50 reaches 85.4% F-measure, 1.7 percentage points above the listed PAN result of 83.7%. With ResNet-101 it reaches 86.6%, 2.9 points above PAN.
On Total-Text, the ResNet-50 version reaches 85.3%, 0.3 points above the listed PAN result of 85.0%; ResNet-101 reaches 87.1%, 2.1 points above PAN. These are strong F-measures on the two arbitrary-shape text benchmarks. The FPS figures across methods use different GPUs, so they are not controlled speed comparisons.
Visualization
Panels (a) through (d) show ICDAR 2013, ICDAR 2015, Total-Text, and CTW-1500 detections. They illustrate text at different scales and shapes.
Conclusion
This paper introduced TextFuseNet, an innovative framework designed for detecting arbitrarily shaped text by combining character-level, word-level, and global semantic features. This multi-level feature fusion approach indeed enhances the model's ability to represent complex text shapes, achieving leading performance on datasets like CTW-1500 and Total-Text.
From the experimental results, TextFuseNet demonstrates strong capabilities in handling intricate text shapes and achieves encouraging results, showcasing the potential of this method.