Real-time Scene Text Detection with Differentiable Binarization
- Publication
- AAAI 2020Conference
- First submitted to arXiv
- arXiv:1911.08947
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textdetectiondbnet2019,
title = {{Real-time Scene Text Detection with Differentiable Binarization}},
author = {Minghui Liao and Zhaoyi Wan and Cong Yao and Kai Chen and Xiang Bai},
year = {2019},
eprint = {1911.08947},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1911.08947}
}
After reviewing several papers on text detection, it's clear that most of the mainstream models are based on segmentation approaches to accomplish the task.
A segmentation mask tells us which pixels are text. The catch is that a single text/background mask cannot tell us where one word or line ends and the next begins: neighboring instances may merge.
With different classes, we could assign each class a channel. But region A and region B are both text. Could we just give the model 1,000 channels and let each instance take one? Perhaps—but who assigns an instance to a channel, and how do we supervise that choice? More channels alone do not solve the matching problem.
And doesn't one high-resolution channel per text instance seem wasteful? Memory and computation would grow quickly, and how many channels should we reserve in the first place? This is about separating adjacent instances, not a general solution to text that physically overlaps or occludes other text.
DBNet keeps the segmentation setup and instead tries to make neighboring text boundaries easier to separate without expensive postprocessing.
Defining the Problem
Some mainstream research approaches attempt to resolve this problem by first identifying the text kernel and then working around it, as shown by the "blue line" in the diagram above.
For instance, PSENet detects text kernels and expands them into complete regions. PAN also starts from kernels, then aggregates neighboring pixels. TextSnake uses a text centerline and geometric predictions to reconstruct text regions.
However, the authors of this paper argue that these methods share a common shortcoming: they are too slow!
Instead, the authors propose that we should let the model learn to separate different text instances by itself, without relying on complicated post-processing.
Solving the Problem
Model Architecture
The architecture itself is not the main focus of this paper, and as far as we can tell, it doesn't present anything particularly novel.
As shown above, the part outlined in red is the Backbone, which in the paper is either ResNet-18 or ResNet-50. The part outlined in green is the FPN, which extracts features at multiple scales.
Once we have the multi-scale feature maps, they are upscaled to the same resolution and concatenated for the final prediction head.
The prediction head consists of two branches:
- The first branch predicts the text region at the pixel level, similar to other segmentation-based methods.
- The second branch, and the focus of this paper, predicts the Threshold Map, providing an adaptive binarization threshold for each pixel.
During training, the probability map and threshold map enter a differentiable binarization function to produce an approximate binary map and help separate adjacent text. Inference can use only the probability map and omit the threshold branch.
This part of the process is not explained in great detail in the paper, so let's take a look at the code implementation: MhLiao/DB.
First, let's look at the text region prediction branch:
self.binarize = nn.Sequential(
nn.Conv2d(inner_channels, inner_channels //
4, 3, padding=1, bias=bias),
BatchNorm2d(inner_channels//4),
nn.ReLU(inplace=True),
nn.ConvTranspose2d(inner_channels//4, inner_channels//4, 2, 2),
BatchNorm2d(inner_channels//4),
nn.ReLU(inplace=True),
nn.ConvTranspose2d(inner_channels//4, 1, 2, 2),
nn.Sigmoid())
Here, the input feature maps from FPN are processed through a series of convolutions and transposed convolutions. The final output is a Sigmoid value between 0 and 1, representing the probability of each pixel being part of a text region.
Next is the threshold map prediction branch:
self.thresh = nn.Sequential(
nn.Conv2d(in_channels, inner_channels //
4, 3, padding=1, bias=bias),
BatchNorm2d(inner_channels//4),
nn.ReLU(inplace=True),
self._init_upsample(inner_channels // 4, inner_channels//4, smooth=smooth, bias=bias),
BatchNorm2d(inner_channels//4),
nn.ReLU(inplace=True),
self._init_upsample(inner_channels // 4, 1, smooth=smooth, bias=bias),
nn.Sigmoid())
Again, input feature maps from FPN go through convolutions and upsampling operations. The final output is a Sigmoid value between 0 and 1, representing the threshold map.
During training, the probability map is compared with the threshold map and passed through a differentiable function to obtain an approximate binary map:
def forward(self, features, gt=None, masks=None, training=False):
# Skip Backbone + Neck code
# Compute text region map
binary = self.binarize(fuse)
if self.training:
result = OrderedDict(binary=binary)
else:
# In inference mode, return only the text region map
return binary
# In training mode
if self.adaptive and self.training:
if self.serial:
fuse = torch.cat(
(fuse, nn.functional.interpolate(
binary, fuse.shape[2:])), 1)
# Compute threshold map
thresh = self.thresh(fuse)
# Apply the step function to threshold the map
thresh_binary = self.step_function(binary, thresh)
# Return both results for supervision during training
result.update(thresh=thresh, thresh_binary=thresh_binary)
return result
# Binarization function
def step_function(self, x, y):
return torch.reciprocal(1 + torch.exp(-self.k * (x - y)))
Now we can finally see why the paper is called a differentiable binarization network! The step_function subtracts the threshold map from the probability map and applies . Its output stays continuous in : almost 0 or 1, yet still differentiable.
This might look familiar:
torch.reciprocal(1 + torch.exp(-self.k * (x - y))) ...
Hey! That's just the Sigmoid function!
The difference here is that the function includes a parameter , which controls the steepness of the binarization. In the paper, this parameter is set to 50.
So, the function becomes:
If you plug in some values, you'll see that with , the function forms a very steep step, making it almost like a binarization function, but still differentiable.
Label Generation
This part of the algorithm is inspired by PSENet:
Given a text image, each text region's polygon is described by a set of lines:
Here is the number of vertices. DBNet's label-generation description says ICDAR 2015 has 4 vertices and CTW1500 has 16. The original CTW1500 dataset paper, however, specifies 14-point polygon annotations, as described in the dataset overview below.
Then, the positive text region is generated by shrinking the polygon into using the Vatti clipping algorithm.
The shrinkage distance is calculated based on the perimeter and area of the original polygon:
Where is the shrinkage ratio, empirically set to 0.4.
A similar process is used to generate labels for the threshold map.
First, the text polygon is expanded to using the same offset . The region between and is considered the text boundary, and within this region, the threshold map label is generated by calculating the distance to the nearest segment in .
Loss Function
The total loss function is defined as the weighted sum of the probability map loss , the binary map loss , and the threshold map loss :
Where is the loss for the probability map, and is the loss for the binary map. Empirically, and are set to 1.0 and 10, respectively.
The paper writes the same binary cross-entropy (BCE) form for the probability map loss and approximate binary map loss . To address the imbalance between positive and negative samples, hard negative mining samples difficult negatives.
The specific loss calculation is as follows:
Here is the sampled set, with a positive:negative ratio of 1:3. The leading minus sign is required for a BCE loss to be minimized; it is omitted in the paper's printed equation. The authors' public loss implementation instead uses BCE for the probability map and Dice Loss for the approximate binary map.
The threshold map loss is the sum of distances between the predicted and labeled threshold values within the expanded text polygon :
Where is the set of pixel indices inside the expanded polygon , and represents the threshold map label.
During inference, to improve efficiency, the authors use the probability map to generate bounding boxes, bypassing the need for the threshold branch.
The bounding box generation involves three steps:
-
Binarizing the probability map, or the approximate binary map if that branch is retained, with a fixed threshold of 0.2.
-
Extracting connected components (shrunk text regions) from the binary map.
-
Expanding these shrunk regions using the Vatti clipping algorithm, with the expansion offset calculated as:
Where is the area of the shrunk polygon, is its perimeter, and is empirically set to 1.5.
Training Datasets
-
SynthText
This dataset is used for pre-training the model. It contains around 800,000 synthetic images where text is blended into natural scenes with random fonts, sizes, colors, and orientations.
-
CTW1500
CTW1500 is a challenging dataset for long, curved text detection, containing 1000 training and 500 test images. Unlike traditional datasets like ICDAR 2015 and ICDAR 2017 MLT, CTW1500 annotates text instances using 14-point polygons, which can describe arbitrary-shaped curved text.
-
Total-Text
Total-Text is a newly released curved text detection dataset, including horizontal, multi-oriented, and curved text instances. It consists of 1255 training images and 300 test images.
-
ICDAR 2015
ICDAR 2015 is a widely-used dataset for text detection, containing 1500 images (1000 for training and 500 for testing). Text regions are annotated with quadrilateral vertices.
-
MSRA-TD500
This dataset includes multi-language, arbitrary-oriented, and long text lines. It contains 300 training and 200 test images, with text lines annotated at the line level. Due to the small size of the training set, images from the HUST-TR400 dataset were added to augment training data.
-
ICDAR 2017 MLT
IC17-MLT is a large-scale multi-lingual text dataset that includes 7,200 training images, 1,800 validation images, and 9,000 test images. It contains full scene images across nine languages, making it a comprehensive dataset for multi-lingual text detection.
Discussion
Ablation Study
From the table above, differentiable binarization improves performance on both MSRA-TD500 and CTW1500. Without deformable convolutions, the ResNet-18 F-score increases by 3.7 and 4.9 percentage points, respectively. Removing the threshold branch at inference leaves the table's FPS unchanged.
Compared with the same backbone without deformable convolution, adding it raises F-score by 1.5 to 5.0 percentage points, with little extra runtime. On CTW1500 without DB, ResNet-18 gains 3.6 percentage points and ResNet-50 gains 4.9 percentage points.
Although unsupervised threshold maps visually resemble the supervised ones, supervision leads to better performance. The table shows that threshold map supervision improves ResNet-18's performance by 0.7 percentage points and ResNet-50's by 2.6 percentage points on the MLT-2017 dataset.
Among backbone networks, ResNet-50 outperforms ResNet-18 but runs at a slower speed.
Curved Text Detection Results
Total-Text Results
CTW1500 Results
The authors report strong accuracy and speed on both Total-Text and CTW1500, which test the model on curved text shapes.
DB-ResNet-50 improves over the previous best scores listed in these tables by 1.1 percentage points on Total-Text and 1.2 percentage points on CTW1500. Its FPS exceeds the listed comparators that report speed; a ResNet-18 backbone further increases speed with a small performance drop.
ICDAR 2015 Results
ICDAR 2015 Results
ICDAR 2015 is a multi-oriented text dataset containing many small and low-resolution text instances.
According to the table, DB-ResNet-50 (1152) has the highest F-score. Compared with the table's previously fastest method, EAST, DB-ResNet-50 (736) scores 7.2 percentage points higher and runs at about twice its FPS.
With the ResNet-18 backbone, DB-ResNet-18 (736) runs at 48 FPS and achieves an F-score of 82.3.
Visualization of Results
For each group of images, the top-right panel shows the predicted threshold map and the bottom-right panel the probability map. The green contours show detections after postprocessing.
Conclusion
DB introduces differentiable binarization during training to improve separation of text regions. Inference can omit the threshold branch, but still uses thresholding, connected components, and polygon expansion to obtain final text detections.
In practice, DB is widely used for deployment primarily due to its speed. Moreover, its simple architecture makes it easy to train. The model has been implemented in various OCR frameworks such as PaddleOCR and MMOCR.
As we have seen, the most challenging part of the paper lies in implementing the binarization function. (Huh?)
The architecture improves separation of neighboring text regions. It is not a general solution to genuine overlap or occlusion between different text instances.