MixNet: Toward Accurate Detection of Challenging Scene Text in the Wild
- Publication
- Publication venue not yet verified
- First submitted to arXiv
- arXiv:2308.12817
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textdetectionmixnet2023,
title = {{MixNet: Toward Accurate Detection of Challenging Scene Text in the Wild}},
author = {Yu-Xiang Zeng and Jun-Wei Hsieh and Xin Li and Ming-Ching Chang},
year = {2023},
eprint = {2308.12817},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2308.12817}
}
As the performance on validation datasets approaches saturation, the key to success lies in the accuracy of the final text control points.
Background noise might be a problem that must be addressed.
Problem Definition
When detecting text in natural scenes, text blocks often appear in arbitrary shapes and orientations. In small and curved text instances, traditional convolutional neural network-based detection methods are easily disturbed by localized noise and variations in image quality, making it difficult to stably and accurately capture the global geometric distribution characteristics of the text.
The authors believe the issue here lies in existing methods focusing on local image features, ignoring the global information about the overall text's geometric distribution. As a result, when facing small, curved text or interference from lighting and background noise, the robustness of detection significantly decreases.
Additionally, in commonly used backbone networks (such as ResNet, VGG, HRNet), when providing high-resolution features, it is difficult to extract clear and easily measurable information due to noise interference, which is especially unfavorable for small text instances.
Therefore, the core problem the authors aim to solve is how to "maintain the intercommunication between high and low-resolution features" to reduce the negative impact of noise interference on high-resolution features.
Solution
Model Architecture
To solve the problem of feature communication, most CNN architectures heavily rely on local convolution operations and lack effective cross-scale feature communication channels, making it difficult to fully utilize the noise suppression ability of low-resolution features.
In this paper, the authors propose a new network structure that combines the Backbone and Neck parts, allowing high- and low-resolution features to interweave and jointly produce more robust, fine, and noise-resistant representations. This structure is called "Feature Shuffle Network," abbreviated as FSNet.
Another part of the system is based on the features generated by FSNet to refine the control points of text instances. This part is called the "Central Transformer Block."
Let’s first look at the structure of FSNet.
Feature Shuffle Network
At first glance, this structure may seem a bit complicated, but it is actually based on a simple concept. Let's break it down:
- Red box part: This corresponds to the feature extraction part of the Backbone network, which includes several stages of downsampling processes.
- Blue box part: This corresponds to the feature extraction part of the Neck network, where the feature shuffle operation is implemented.
The method is to split the feature channels of each resolution, remix them, upsample/downsample, and finally concatenate them into new features.
Through this mixing and reshuffling, the model can more efficiently leverage the advantages of features at different scales, producing more recognizable features for small, curved, and low-quality text. The detailed operation is shown in the figure below:
In FSNet, two shuffle layers are designed. The first shuffle layer receives input features from two scales, while the second shuffle layer receives input features from three scales.
Taking the second shuffle layer as an example, suppose we have three sets of input features, , where represents the feature at the th resolution. The shuffle layer splits the channels of each feature into three equal parts, i.e., is divided into .
Then, based on the resolution index corresponding to each part, the features are upsampled or downsampled to a uniform size. Finally, the feature parts from different resolutions and re-scaled at different rates are concatenated into a new feature representation.
In this way, the original single-resolution features are "shuffled" into new features that mix multiple resolution information.
Central Transformer Block
Let's break down the architecture diagram and isolate the CTBlock part to examine its operation flow:
-
Contour Points:
Initially, CTBlock uses the heatmap generated by FSNet to obtain a rough boundary of each text instance. Then, it samples N evenly spaced points (with N=20 in the experiment) along the boundary as the text contour representative points. The image features of these points, along with the corresponding heatmap values, form a feature sequence that is fed into the first Transformer module.
-
Central Points:
The first Transformer module generates center-line points from the contour features (the number of center-line points is fixed at C=10). The image features and heatmap values at these points form another feature sequence. During training, the center-line points are supervised using the annotated upper and lower text contours.
-
Transformer Encoder and MLP Decoder:
CTBlock uses Transformer encoders with self-attention and MLP blocks to model the overall geometry of the sampled points. The second Transformer module combines the contour and center-line feature sequences; its simple MLP decoder predicts a vertex offset for each contour point. Adding these offsets to the initial contour produces the refined text boundary.
The correspondence between contour and central points is shown in the figure below:
Implementation Details
- Backbone: FSNet
- Pre-training: The model is trained for 5 epochs on the SynthText dataset using the Adam optimizer, with a fixed learning rate of 0.001.
- Fine-tuning: The model is trained for 600 epochs on Total-Text, ArT, and other datasets.
- Training Environment: Python3, PyTorch 1.10.2, RTX-3090 GPU
- Input Image Size: 640 × 640
- Data Augmentation: Random rotation (−30° to 30°), random cropping, random flipping, color jittering
Discussion
Effectiveness of FSNet
The main focus of this paper is FSNet, so let’s first see if FSNet actually works.
The authors first classify the text into small, medium, and large categories based on size and compare the detection performance of FSNet with other commonly used backbone networks (such as ResNet50) on a noise-free dataset.
On the noise-free test set, FSNet raises the detection score for small text from 53.0% to 68.8%, and for medium text from 88.0% to 92.4%. The authors attribute this improvement to the exchange of high- and low-resolution features in FSNet.
In addition, to validate FSNet's robustness against noise, the authors add specific ratios of impulse noise to the test data and compare the detection performance of FSNet and ResNet50 under noise-free, 5% noise, and 10% noise conditions.
Although noise degrades the performance of all models, FSNet still demonstrates better robustness. In the noise-free case, FSNet's F1-score is 2.1 percentage points higher than ResNet50, and under 5% and 10% noise, the improvement reaches 4.5 and 6.0 percentage points, respectively.
Finally, the authors replace both the backbone and FPN architectures of existing text detection methods, including DB, PAN, and FAST, with FSNet and evaluate them on Total-Text. The results are shown below:
Under this replacement setup, the F1-scores of DB and PAN each rise by 1.5 percentage points, and FAST rises by 1.8 percentage points. This shows that FSNet can improve these three existing detectors on Total-Text.
Ablation Study
Next, the authors validate the effectiveness of each design. Here, we see the results of an ablation study using the Total-Text dataset, with ResNet50 as the baseline model for comparison.
With the same SynthText pretraining, F1 rises from 87.7% for the ResNet50 baseline to 89.9% with FSNet (a 2.2-point gain), then to 90.5% with center-line features (another 0.6 points). Without pretraining, the corresponding gains are 1.6 and 0.5 percentage points.
Comparison with Other Methods
The table above compares multi-oriented text detection on MSRA-TD500. MixNet reaches an F1-score of 89.4%, 2.1 percentage points above the next highest result, FAST at 87.3%. These are results for multilingual, multi-oriented text under the MSRA-TD500 test conditions.
Next, we look at the comparison for arbitrary-shaped text detection on the Total-Text dataset, as shown in the table above.
MixNet reaches an F1-score of 90.5% on Total-Text, 1.5 percentage points above the next highest result, DPText-DETR at 89.0%. Compared with TextFuseNet at 87.5%, the gap is 3.0 percentage points.
Visualization Results
Conclusion
The MixNet architecture proposed in this paper, with FSNet and the CTBlock modules as the core, reduces the problem of high-resolution features being easily affected by noise during the traditional multi-scale feature fusion process. It also significantly improves the detection accuracy for small, curved, and low-quality text.
The experimental results also confirm that FSNet can be easily integrated into different architectures and effectively improve their overall detection performance.