Back to paper library
arXiv 2021IEEE TPAMI 2021Text spottingPaper notes

ABCNet v2

The One-bit Challenge

Original paper

ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting

Paper authors

Yuliang Liu · Chunhua Shen · Lianwen Jin …

Show all 7 authors

Yuliang Liu · Chunhua Shen · Lianwen Jin · Tong He · Peng Chen · Chongyu Liu · Hao Chen

First submitted to arXiv
arXiv:2105.03620
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textspottingabcnetv22021,
  title = {{ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting}},
  author = {Yuliang Liu and Chunhua Shen and Lianwen Jin and Tong He and Peng Chen and Chongyu Liu and Hao Chen},
  year = {2021},
  eprint = {2105.03620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.03620}
}
On this page

After the first ABCNet paper, the authors revised and expanded the work into ABCNet v2. Much of the core approach and even the training setup carries over, so we'll focus on what was added. For the basics, see our previous note:

Problem Definition

Inspired by several other works, the authors decided to enhance ABCNet. Their main references for this upgrade were:

  1. [19.11] EfficientDet: Scalable and Efficient Object Detection

    • This paper introduces the BiFPN, an optimized version of the Feature Pyramid Network (FPN). The authors adopted this approach in ABCNet.
  2. [18.07] An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

    • This paper proposes CoordConv, highlighting the importance of spatial coordinate information for CNNs.
  3. [20.03] SOLOv2: Dynamic and Fast Instance Segmentation

    • SOLOv2 implements CoordConv for instance segmentation, which the authors also integrated into ABCNet.

The upgraded ABCNet v2 introduces several innovations:

  • Bezier curve-based text representation. (This feature remains from the first version.)
  • BezierAlign feature alignment method. (Also retained from the original.)
  • BiFPN implementation, a bidirectional multi-scale feature pyramid for global text features. (New addition.)
  • Adaptive end-to-end training strategy. (New addition.)
  • Addition of a Chinese text recognition dataset, enabling Chinese character recognition. (New addition.)
  • Model quantization to enhance inference speed. (New addition.)

Problem-Solving Strategies

Let’s review the newly introduced components in ABCNet v2.

Model Architecture

Coordinate Convolution Module

This module, known as CoordConv, addresses the limitations of traditional convolutional layers when mapping (x,y)(x,y) coordinates to pixel space. The authors concatenate two channels containing the absolute xx and yy coordinates at each position with the FPN feature maps, producing fcoordf_{coord}. They then pass these features through three convolutional layers (kernel size 3, stride 1, padding 1) to improve scene text detection.

Attention-based Recognition Branch

The recognition branch consists of six convolutional layers, one bidirectional LSTM, and an attention module. During recognition, the model gradually predicts characters by using the previous hidden state and weighted sums. The character classes are set to 96 (English letters, digits, and symbols, excluding EOS) or 5462 (the bilingual Chinese–English inventory, excluding EOS). The attention weights are computed as follows:

et,s=K⊤tanh⁡(Wht−1+Uhs+b)e_{t,s} = K^\top \tanh(W h_{t-1} + U h_s + b) at,s=exp⁡(et,s)∑r=1nexp⁡(et,r)a_{t,s}=\frac{\exp(e_{t,s})}{\sum_{r=1}^{n}\exp(e_{t,r})}

Weighted sum:

ct=∑s=1nat,shsc_t = \sum_{s=1}^n a_{t,s} h_s

Updating the hidden state:

ht=GRU((embedt−1,ct),ht−1)h_t = \text{GRU}((\text{embed}_{t-1}, c_t), h_{t-1})

Adaptive End-to-End Training (AET)

In ABCNet’s first version, only ground-truth Bezier curves were used to align features for the recognition branch during training, while detection results were used for cropping at test time. The authors observed that errors could arise when the detected curves were less precise than the annotated boundaries.

To address this, the authors proposed Adaptive End-to-End Training (AET), which operates as follows:

  1. Detection Result Filtering: Detection results are first filtered by confidence thresholds, and Non-Maximum Suppression (NMS) is applied to remove redundant detections.

  2. Recognition Label Assignment: Each detection result is assigned recognition labels by minimizing the sum of distances between the detected control points and the ground-truth control points. The formula is as follows:

    j∗=arg⁡min⁡j∑i=1n∥cpi−cpj,i∗∥1,rec=recj∗∗j^* = \arg\min_j \sum_{i=1}^{n}\|cp_i-cp^*_{j,i}\|_1, \qquad \text{rec}=\text{rec}^*_{j^*}

    Here jj indexes ground-truth text instances, ii indexes the nn control points, and ∥⋅∥1\|\cdot\|_1 sums coordinate differences. Recognition uses the transcription of the closest instance j∗j^*. This is a geometric assignment rule.

  3. Label Set Update: The new target labels are concatenated with the original ground-truth set, which is then used for further recognition training.

Model Quantization

The authors explore model quantization, an approach to reduce model precision to improve inference speed—an area not commonly studied in text detection.

Discussion

Ablation Study

To evaluate the impact of each component, the authors conducted ablation studies on the Total-Text and SCUT-CTW1500 datasets. The main results are as follows:

  1. Attention-based Recognition Module: Relative to the CTC baseline, attention raises end-to-end Hmean from 63.5% to 66.2% on Total-Text (2.7 percentage points) and from 45.0% to 52.9% on SCUT-CTW1500 (7.9 points).
  2. BiFPN Architecture: Starting from the attention-only setting, BiFPN adds 1.9 and 1.6 percentage points, respectively; the table's speed falls from 11 to 10 FPS.
  3. Coordinate Convolution (CoordConv): Also relative to the attention-only setting, CoordConv adds 2.8 and 2.9 percentage points, respectively, while the listed speed remains 11 FPS.
  4. Adaptive End-to-End Training (AET): Again relative to attention only, AET adds 1.2 and 1.7 percentage points, respectively.
  5. Bezier Curve Order: With labels regenerated for fourth-order curves and the model trained to regress their control points, the attention-only results rise from 66.2% to 67.0% on Total-Text and from 52.9% to 54.4% on SCUT-CTW1500. With fifth-order curves on Total-Text, Hmean instead falls to 65.5%. The authors suggest that excessively high order may make control points vary too much and harder to regress.

Comparison of Text Detection Methods

Experiments on the detection task were conducted across five datasets, including two for arbitrarily-shaped text (Total-Text and SCUT-CTW1500), two for multi-oriented text (MSRA-TD500 and ICDAR 2015), and a bilingual dataset, ReCTS.

By the table's Hmean columns, ABCNet v2 has the highest listed score on all five datasets: for example, 87.0% on Total-Text, 84.7% on SCUT-CTW1500, and 90.4% on ReCTS.

Comparison of Text Recognition Methods

For end-to-end spotting on SCUT-CTW1500, ABCNet v2 scores 57.5%/77.2% without/with the full lexicon, or 58.4%/79.0% at multiple scales, all highest in the table. On ICDAR 2015, its single-scale strong/weak lexicon scores, 82.7%/78.5%, trail Feng et al.'s 87.3%/83.1%; its multi-scale generic-lexicon score of 75.0% is highest. On ReCTS, its 1-NED of 62.7% (65.4% at multiple scales) trails Mask TextSpotter’19's 67.8%.

Quantized Model Performance

“A/W” represent activation and weight bit-width configurations, respectively, while "†" indicates the use of a progressive training strategy.


The authors conducted an extensive analysis on the performance of quantized models.

Initially, they pre-trained the low-bit models on a synthetic dataset and then fine-tuned them on TotalText and CTW1500 for improved performance. The pre-training phase involved 260K iterations with a batch size of 8, starting with a learning rate of 0.01, which was reduced tenfold at the 160K and 220K iterations. For fine-tuning on TotalText, the batch size remained 8, with an initial learning rate of 0.001, and 5K iterations were conducted. The same configuration was used for fine-tuning on CTW1500, but with a total of 120K iterations, and the learning rate was reduced tenfold at the 80K iteration. All convolutional layers in the network were quantized, except for the input and output layers; the LSTM remains full precision.

The results, shown in the table, highlight the following observations:

  • 4-bit Models: The performance of 4-bit models is nearly equivalent to that of full-precision models.

    • After pretraining on the synthetic dataset, the 4-bit model's end-to-end Hmean is 58.9%, above the full-precision model's 57.6%. The paper's prose reverses these two numbers; the table shows this order.
    • After fine-tuning on TotalText and CTW1500, the 4-bit versus full-precision Hmeans are 66.5% versus 67.2% and 52.2% versus 53.0%, lower by 0.7 and 0.8 percentage points.

    In this ResNet-18 experiment, where convolutional layers are mainly quantized, the 4-bit model remains close to full precision. The listed FPS comes from profiling on a single 2080 Ti.

  • 1-bit Models (Binary Network, BNN): Without progressive training, the binary network reaches only 44.3% end-to-end Hmean after pretraining, a notable drop.

    • To address this, the authors proposed a progressive training strategy (indicated by †), gradually reducing the quantization bit-width (e.g., from 4-bit → 2-bit → 1-bit), which significantly improved BNN performance.
    • With the new strategy, the BNN pretrained on the synthetic dataset reaches 55.8% end-to-end Hmean, 1.8 percentage points below the full-precision version's 57.6% at the same stage.

Conclusion

This paper introduced several new components, including the attention-based recognition module, BiFPN structure, coordinate convolution, and a novel adaptive end-to-end training strategy.

The authors showcased a variety of training techniques and strategies, offering valuable insights with practical applications. This work is highly instructive, presenting many techniques that could be beneficial in real-world implementations.