Back to paper library
arXiv 2019AAAI 2020Text recognitionPaper notes

DAN

They Cannot Be Together

Original paper

Decoupled Attention Network for Text Recognition

Paper authors

Tianwei Wang · Yuanzhi Zhu · Lianwen Jin …

Show all 8 authors

Tianwei Wang · Yuanzhi Zhu · Lianwen Jin · Canjie Luo · Xiaoxue Chen · Yaqiang Wu · Qianying Wang · Mingxiang Cai

Publication
AAAI 2020Conference
First submitted to arXiv
arXiv:1912.10205
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitiondan2019,
  title = {{Decoupled Attention Network for Text Recognition}},
  author = {Tianwei Wang and Yuanzhi Zhu and Lianwen Jin and Canjie Luo and Xiaoxue Chen and Yaqiang Wu and Qianying Wang and Mingxiang Cai},
  year = {2019},
  eprint = {1912.10205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.10205}
}
On this page

Earlier work mostly used CTC to transcribe the final output. More recently, architectures with attention maps became popular. The reason is simple: they often work better!

So adding an attention map had become almost standard equipment in scene text recognition.

Problem Definition

But attention maps can go astray!

Attention Loss

The authors identify a problem in conventional attention: alignment depends jointly on visual features and the decoder state, so recognition errors can shift subsequent attention.

What does that mean? Let's take a look at the image below:

In the image above, the top row shows the original input image, while the bottom row displays the corresponding attention map.

These attention maps are structured with sequential information: the top-left corner corresponds to the first character, and the bottom-right corner corresponds to the last one. A good attention distribution should have a diagonal pattern, indicating that each character is correctly aligned with a specific region.

In the example, however, the distribution deviates from the diagonal and shifts to another region. This suggests that the model fails to focus on the correct textual information at certain time steps.

Similar local features can make alignment ambiguous. In the example, features around “ly” resemble those at other sequence positions, and the decoder may attend to the wrong region.

Coupled Output

We need attention maps to align text and images effectively. However, if alignment and decoding operations are coupled, alignment is influenced by the decoder state. This coupling can allow recognition errors to propagate to later alignment decisions.


Thus, the authors argue: alignment and decoding should not be coupled.

Solution

Model Architecture

To address the coupling issue, the authors propose the Decoupled Attention Network (DAN) architecture, as shown above. The key idea is to design a separate branch dedicated solely to learning attention and predicting the character positions.

Let’s break down the architecture step-by-step:

Feature Extraction

The first step involves a CNN network to extract features from the input image, which is a common approach.

The authors use a CNN-based feature encoder similar to previous works, instead of a standard ResNet. The configuration is as follows:

The input image xx (of size H×WH \times W) is encoded into a feature map FF:

F=F(x),F∈RC×H/rh×W/rwF = F(x), \quad F \in \mathbb{R}^{C \times H/r_h \times W/r_w}

where CC is the number of output channels, and rhr_h and rwr_w are down-sampling factors for height and width, respectively.

Alignment Module

The Convolutional Alignment Module (CAM) processes the feature maps.

CAM collects multi-scale visual features from the feature encoder and processes them through a series of down-sampling convolution layers. Inspired by Fully Convolutional Networks (FCNs), CAM computes attention for each channel, with each output channel representing one decoding position’s attention map, not a character class.

CAM contains LL layers. During the upsampling stage (using transposed convolutions), the output features from each layer are added to the corresponding down-sampled feature map. Finally, sigmoid activation and per-channel normalization generate the attention map:

A={α1,α2,…,αmaxT}A = \{ \alpha_1, \alpha_2, \dots, \alpha_{\text{maxT}} \}

where maxT\text{maxT} is the maximum sequence length for decoding. Each attention map has the size H/rh×W/rwH/r_h \times W/r_w.

At each decoding step, a corresponding attention map weights and aggregates the shared visual feature map into the context for that character.

Decoupled Decoder Module

Unlike conventional attention decoders, which handle both alignment and recognition, DAN’s decoder only handles recognition.

Given the encoded feature map and the attention map, the decoder computes the context vector ctc_t at time step tt:

ct=∑x=1W/rw∑y=1H/rhαt,x,yFx,yc_t = \sum_{x=1}^{W/r_w} \sum_{y=1}^{H/r_h} \alpha_{t, x, y} F_{x, y}

The classifier generates the output yty_t at each time step:

yt=wht+by_t = w h_t + b

where hth_t is the hidden state of a GRU:

ht=GRU((et−1,ct),ht−1)h_t = \text{GRU}((e_{t-1}, c_t), h_{t-1})

et−1e_{t-1} is the embedding of the previous decoding result yt−1y_{t-1}. During training, the previous ground-truth label is supplied as this input.

The loss function for DAN is:

Loss=−∑t=1Tlog⁡P(gt∣I,θ)\text{Loss} = -\sum_{t=1}^{T} \log P(g_t | I, \theta)

where θ\theta represents all trainable parameters, and gtg_t is the ground truth label at step tt.

Handwriting Recognition Training Strategy

Datasets

  1. IAM Dataset:

    • English handwritten text based on the LOB corpus.
    • Training: 747 documents (6,482 lines).
    • Validation: 116 documents (976 lines).
    • Test: 336 documents (2,915 lines).
  2. RIMES Dataset:

    • Handwritten French letters.
    • Training: 1,500 paragraphs (11,333 lines).
    • Test: 100 paragraphs (778 lines).

The model is trained on whole text lines from both datasets, using an open-source data-augmentation tool. Input height is normalized to 192, with width scaled in proportion to the original image (up to 2048). A 3×13 \times 1 convolution at the end of the feature encoder reduces the feature map to 1D.

  • maxTmaxT is set to 150 to cover the longest text lines.
  • CAM layers (except the last) have 128 channels.

At test time on RIMES, the authors try six predefined crops. For example, {10,10} removes ten rows from both the top and bottom. They recognize those crops and the original image, average the output probabilities for each result, and select the highest-scoring one.

The model uses Character Error Rate (CER%) and Word Error Rate (WER%) for evaluation:

CER / WER=Edit DistanceGround-truth Characters / Words\text{CER / WER} = \frac{\text{Edit Distance}}{\text{Ground-truth Characters / Words}}

No external language model or lexicon is used in these experiments.

Scene Text Recognition Training Strategy

Datasets

  1. Regular Scene Text:

    • IIIT5K-Words: 3,000 cropped word images.
    • Street View Text (SVT): 647 word images.
    • ICDAR 2003 (IC03): 867 word images.
    • ICDAR 2013 (IC13): 1,015 word images.
  2. Irregular Scene Text:

    • SVT-Perspective (SVT-P): 639 perspective images.
    • CUTE80: 288 curved-text images.
    • ICDAR 2015 (IC15): 2,077 blurred and multi-oriented images.

Training data comes from the MJSynth (Synth90k) and SynthText synthetic text datasets. Input height is set to 32, with width scaled proportionally (up to 128). Other settings include:

  • maxT=25maxT = 25 (maximum sequence length).
  • CAM depth L=8L = 8 with 64 channels per layer (except the last).
  • Bidirectional decoder used for final predictions.
  • ADADELTA optimizer with an initial learning rate of 1.0, reduced to 0.1 after the third epoch.

Discussion

Handwritten Text Recognition Performance

  • IAM Dataset:

    • DAN has the lowest CER in the table, reducing it by 1.5 percentage points, from 7.9% to 6.4%.
    • WER: Although Bhunia et al. (2019) perform better in WER, their model requires cropped single-word images, whereas DAN can recognize entire lines of text, offering greater flexibility in real-world applications.
  • RIMES Dataset:

    • CER: DAN's 2.7% is 0.2 percentage points lower than the previous table best of 2.9%.
    • WER: DAN reduces it by 3.7 percentage points, from 12.6% to 8.9% (about 29% relative error reduction).
    • The authors take the WER improvement as a sign that the model learns useful semantic information for long-text recognition.

Ablation Study

At the tested output lengths of 150, 200, and 250, WER and CER are almost unchanged. Forward time on a TITAN X GPU rises from 188.7 to 190.5 ms per iteration. In this experiment, performance changes little as long as the output limit is set beyond the text length.

As the depth LL decreases, DAN’s performance drops significantly, indicating that CAM requires sufficient depth to achieve good alignment. To correctly align a character, CAM's receptive field must be large enough to cover the character and its neighboring regions.

Alignment Error Analysis

The authors track attention centers, identifying the region with the highest attention score. If the current attention center shifts left compared to the previous one, it is recorded as a misalignment. The test samples are divided into five groups based on text length: [0, 30), [30, 40), [40, 50), [50, 60), and [60, 70), with each group containing over 100 samples. The total misalignment data is summed for each group, and the average misalignment per image (MM/img) is calculated.

The trends in CER improvement and reduced misalignment broadly move together. That supports the authors' explanation that fewer alignment errors help recognition, although the graph alone does not isolate every cause of the gain.

The following image demonstrates how DAN resolves misalignment errors. The top row shows the input image, the middle row shows the alignment results using traditional attention, and the bottom row shows the results using DAN.

Recognition Error Analysis

The authors present several error cases for DAN, as shown above:

  1. (a): The character “p” is incorrectly recognized as “e” because the handwriting style makes them hard to distinguish. Even a human may struggle without context.
  2. (b): The recognizer skips a space between two words due to their proximity.
  3. (c): Some noise in the texture is misinterpreted as text by DAN.

Despite these mistakes, DAN demonstrates stronger robustness compared to traditional attention mechanisms. In the presence of noise, traditional attention often produces unpredictable errors because alignment is disrupted, whereas DAN maintains stable alignment, even when generating additional output.

Scene Text Recognition Performance

  • Regular Scene Text Recognition:

    • DAN achieves leading performance in the paper’s comparison on IIIT5K and IC03.
    • On SVT and IC13, DAN’s performance is comparable to the SOTA models, with minor differences.
    • DAN-1D performs better on IC03 and IC13, as the images in these datasets are relatively clean and regular.
  • Irregular Scene Text Recognition:

    • DAN-2D achieves SOTA performance on SVT-P and CUTE80, performing best among 2D models.

Robustness Study

The authors conducted perturbation tests on IIIT5K and IC13 datasets, comparing DAN with CA-FCN.

The perturbation strategies used are as follows:

  1. IIIT-p: Repeating border pixels to pad 10% of the image height and width vertically and horizontally.
  2. IIIT-r-p:
    • Randomly enlarging the four corners of the image (up to 20%).
    • Repeating border pixels to fill the quadrilateral, then transforming it back into an axis-aligned rectangle.
  3. IC13-ex: Expanding the text bounding box by 10% before cropping.
  4. IC13-r-ex: Randomly expanding the box along height and width by up to 20% to form a quadrilateral, then cropping its axis-aligned bounding rectangle.

Judged by the relative accuracy drop, DAN-1D is more stable on IIIT-p and IC13-ex; DAN-2D is also slightly better on IC13-r-ex. Both DAN variants are less stable than CA-FCN on IIIT-r-p.

Conclusion

From the outset, the authors clearly identified the limitations of traditional attention mechanisms and proposed a decoupled solution.

DAN proves to be highly valuable in text recognition tasks, offering flexibility, robustness, and strong performance. Its ability to reduce alignment errors makes it an effective tool. While there is still room for improvement in handling noisy and visually similar text, its simple architecture and dictionary-free design make DAN a strong candidate for various text recognition applications.