Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion
- Publication
- IEEE Transactions on Pattern Analysis and Machine Intelligence 2023Journal
- First submitted to arXiv
- arXiv:2202.10304
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textdetectiondbnetpulspuls2022,
title = {{Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion}},
author = {Minghui Liao and Zhisheng Zou and Zhaoyi Wan and Cong Yao and Xiang Bai},
year = {2022},
eprint = {2202.10304},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2202.10304}
}
When writing a literature review of the DBNet series, one question we’ve had is whether to refer to it as DB or DBNet. In the first version of the paper, there was no mention of DBNet, so other research papers simply referred to it as DB. However, in this latest paper, the authors explicitly call their model DBNet, so we will honor their naming convention and refer to this series as DBNet.
Problem Definition
Following up on the previous DBNet paper, the authors recognized room for further improvement in the model. They proposed DBNet++, which builds upon the existing architecture by optimizing the way features are fused.
If you haven’t read the previous paper, we recommend doing so, as many of the repeated details are omitted here.
Problem Solution
Model Architecture
The overall architecture is almost identical to the first version, so we’ll skip over it. Let’s look directly at this paper’s change: the Adaptive Scale Fusion Module highlighted above.
Adaptive Scale Fusion
The Adaptive Scale Fusion (ASF) module, as the authors refer to it, is a new module built upon the foundation of DBNet.
In the original design, feature maps at different scales were simply concatenated and passed through a convolution layer before the prediction branch. The authors thought this could be more precise: different scales may matter more at different image positions, so the fusion stage should calculate a weight for each scale at each position.
Let’s follow the authors’ process step by step as shown in the image:
- Resize the four feature maps to the same resolution, concatenate them, and apply a convolution to obtain an intermediate map of size
C × H × W. - In Spatial Attention:
- Average over channels, preserving each spatial position, to obtain
1 × H × W. - Apply
Conv-ReLUandConv-Sigmoid, then expand the result back toCchannels. - Add it to the intermediate map and apply another
Conv-Sigmoidto obtain one spatial weight map per scale:N × 1 × H × W, whereN = 4.
- Average over channels, preserving each spatial position, to obtain
- Back in Adaptive Scale Fusion:
- Expand each scale’s weight across
Cchannels and multiply it by that scale’s input feature map. - Concatenate the four weighted maps along the channel dimension to obtain
(N × C) × H × W.
- Expand each scale’s weight across
Discussion
Comparison with the Original DBNet
The authors conducted ablation experiments on the MSRA-TD500 and CTW1500 datasets, testing modules such as Differentiable Binarization (DB), Deformable Convolution, and Adaptive Scale Fusion (ASF).
Let’s focus on the red boxes, which show the effect of adding ASF.
In the highlighted ResNet-50 rows, both deformable convolution and DB are enabled. Adding ASF leaves MSRA-TD500 precision at 91.5, while recall rises from 79.2 to 83.3. On CTW1500, precision rises from 86.9 to 87.9 and recall from 80.2 to 82.8. In the other settings, precision moves in both directions; false-positive counts require separate inspection.
With ResNet-50, DB and deformable convolution held fixed, adding ASF increases F-measure from 84.9 to 87.2 on MSRA-TD500 (2.3 percentage points) and from 83.4 to 85.3 on CTW1500 (1.9 percentage points). The corresponding speeds drop from 32 to 29 FPS and 22 to 21 FPS. These rows support ASF’s benefit on both datasets under this setting.
Visualization
In each example, the text detection result is on the left, the threshold map is at the upper right, and the probability map is at the lower right. These are examples of DB outputs; this figure alone cannot quantify ASF’s gain.
Conclusion
Much of the content in this paper is similar to the previous DBNet, so we won’t go over it in great detail again.
The Adaptive Scale Fusion module lets the model adjust how strongly it uses features at each scale and position. In the ResNet-50 ablation above, this improves F-measure by roughly two percentage points while modestly reducing inference speed.
All told, that looks like a pretty good investment!