Back to paper library
arXiv 2022IEEE TPAMI 2023Text detectionPaper notes

DBNet++

Adaptive Feature Fusion

Original paper

Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion

Paper authors

Minghui Liao · Zhisheng Zou · Zhaoyi Wan · Cong Yao · Xiang Bai

First submitted to arXiv
arXiv:2202.10304
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textdetectiondbnetpulspuls2022,
  title = {{Real-Time Scene Text Detection with Differentiable Binarization and Adaptive Scale Fusion}},
  author = {Minghui Liao and Zhisheng Zou and Zhaoyi Wan and Cong Yao and Xiang Bai},
  year = {2022},
  eprint = {2202.10304},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.10304}
}
On this page

When writing a literature review of the DBNet series, one question we’ve had is whether to refer to it as DB or DBNet. In the first version of the paper, there was no mention of DBNet, so other research papers simply referred to it as DB. However, in this latest paper, the authors explicitly call their model DBNet, so we will honor their naming convention and refer to this series as DBNet.

Problem Definition

Following up on the previous DBNet paper, the authors recognized room for further improvement in the model. They proposed DBNet++, which builds upon the existing architecture by optimizing the way features are fused.

If you haven’t read the previous paper, we recommend doing so, as many of the repeated details are omitted here.

Problem Solution

Model Architecture

Liao et al., DBNet++, Figure 3, arXiv:2202.10304v1

The overall architecture is almost identical to the first version, so we’ll skip over it. Let’s look directly at this paper’s change: the Adaptive Scale Fusion Module highlighted above.

Adaptive Scale Fusion

Liao et al., DBNet++, Figure 4, arXiv:2202.10304v1

The Adaptive Scale Fusion (ASF) module, as the authors refer to it, is a new module built upon the foundation of DBNet.

In the original design, feature maps at different scales were simply concatenated and passed through a convolution layer before the prediction branch. The authors thought this could be more precise: different scales may matter more at different image positions, so the fusion stage should calculate a weight for each scale at each position.

Let’s follow the authors’ process step by step as shown in the image:

  1. Resize the four feature maps to the same resolution, concatenate them, and apply a convolution to obtain an intermediate map of size C × H × W.
  2. In Spatial Attention:
    1. Average over channels, preserving each spatial position, to obtain 1 × H × W.
    2. Apply Conv-ReLU and Conv-Sigmoid, then expand the result back to C channels.
    3. Add it to the intermediate map and apply another Conv-Sigmoid to obtain one spatial weight map per scale: N × 1 × H × W, where N = 4.
  3. Back in Adaptive Scale Fusion:
    1. Expand each scale’s weight across C channels and multiply it by that scale’s input feature map.
    2. Concatenate the four weighted maps along the channel dimension to obtain (N × C) × H × W.

Discussion

Comparison with the Original DBNet

Liao et al., DBNet++, Table I, arXiv:2202.10304v1

The authors conducted ablation experiments on the MSRA-TD500 and CTW1500 datasets, testing modules such as Differentiable Binarization (DB), Deformable Convolution, and Adaptive Scale Fusion (ASF).

Let’s focus on the red boxes, which show the effect of adding ASF.

In the highlighted ResNet-50 rows, both deformable convolution and DB are enabled. Adding ASF leaves MSRA-TD500 precision at 91.5, while recall rises from 79.2 to 83.3. On CTW1500, precision rises from 86.9 to 87.9 and recall from 80.2 to 82.8. In the other settings, precision moves in both directions; false-positive counts require separate inspection.

With ResNet-50, DB and deformable convolution held fixed, adding ASF increases F-measure from 84.9 to 87.2 on MSRA-TD500 (2.3 percentage points) and from 83.4 to 85.3 on CTW1500 (1.9 percentage points). The corresponding speeds drop from 32 to 29 FPS and 22 to 21 FPS. These rows support ASF’s benefit on both datasets under this setting.

Visualization

Liao et al., DBNet++, Figure 9, arXiv:2202.10304v1

In each example, the text detection result is on the left, the threshold map is at the upper right, and the probability map is at the lower right. These are examples of DB outputs; this figure alone cannot quantify ASF’s gain.

Conclusion

Much of the content in this paper is similar to the previous DBNet, so we won’t go over it in great detail again.

The Adaptive Scale Fusion module lets the model adjust how strongly it uses features at each scale and position. In the ResNet-50 ablation above, this improves F-measure by roughly two percentage points while modestly reducing inference speed.

All told, that looks like a pretty good investment!