Back to paper library
arXiv 2019CVPR Workshops 2020Text recognitionPaper notes

SATRN

Transformer Reaches the Battlefield

Original paper

On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention

Paper authors

Junyeop Lee · Sungrae Park · Jeonghun Baek · Seong Joon Oh · Seonghyeon Kim · Hwalsuk Lee

Publication
CVPR Workshops 2020Workshop
First submitted to arXiv
arXiv:1910.04396
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionsatrn2019,
  title = {{On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention}},
  author = {Junyeop Lee and Sungrae Park and Jeonghun Baek and Seong Joon Oh and Seonghyeon Kim and Hwalsuk Lee},
  year = {2019},
  eprint = {1910.04396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.04396}
}
On this page

Since the Transformer was introduced, its influence has gradually spread across various fields.

This paper, published by Clova AI, aims to bring the advantages of Transformers to the field of Scene Text Recognition (STR), potentially addressing some of its challenges.

Defining the Problem

The recognition of arbitrarily shaped text continues to be a challenge for researchers.

Although several solutions have been proposed over the past few years, these methods still fall short of delivering ideal results. Most notably, many of these studies assume that the text is arranged horizontally, limiting the model's generalization ability.

By this time, the Transformer had already been around for more than two years, and BERT had taken the NLP world by storm. The authors thought: maybe the Transformer’s attention mechanism could help solve this problem in STR as well.

Solving the Problem

Model Architecture

The authors proposed a model architecture that looks like a standard Transformer but includes some modifications tailored to the STR task. Since the Transformer architecture has been discussed widely, we won’t go into too much detail here. If you are unfamiliar with it, you can refer to:

In the encoder, the input image first passes through a shallow convolutional neural network (CNN) to extract local image patterns and textures, easing the computational load on self-attention. This CNN has two 3×33 \times 3 convolutional layers, each followed by a 2×22 \times 2 max pooling layer with a stride of 2.

The self-attention module is based on Transformer self-attention, utilizing adaptive 2D positional encoding to handle the challenge of arbitrary text arrangements.

For the feedforward network (FFN), the authors introduce a locality-aware layer to capture local structure. It adds a 3×33 \times 3 depthwise convolution to the original pointwise feedforward design while retaining the 1×11 \times 1 convolutions for channel processing.

In the decoder, the text sequence is extracted from the 2D feature map produced by the encoder.

During decoding, the cross-modal interaction between the image input and text output is realized in the second multi-head attention module. The decoder structure largely mirrors that of a Transformer decoder, relying mainly on multi-head attention and pointwise feedforward layers.

Discussion

Comparison with Previous Methods

In Table 1, SATRN scores 79.0/86.5/87.8% on the three irregular-text benchmarks IC15/SVTP/CT80. Compared with the highest earlier result in each column, 76.9/79.6/83.3%, those are gains of 2.1/6.9/4.5 percentage points. The paper's experiments section says 4.7 points on average, while its abstract and conclusion say 5.7 points. Because these statements conflict, the per-dataset results are clearer. Earlier methods in the table also use different training datasets.

The architecture does well on arbitrary text shapes and outperforms the other 2D feature-map methods wherever they report results. On the regular IIIT5K and SVT datasets, however, the STN-based ASTER still scores higher than SATRN.

Comparison with the SAR Model

SATRN differs significantly from the SAR model in both the encoder (self-attention vs convolutional network) and decoder (self-attention vs LSTM). While SAR uses a ResNet encoder and LSTM decoder, SATRN adopts self-attention-based encoders and decoders.

Replacing only SAR's ResNet encoder with the SATRN encoder raises accuracy by 1.0/0.9 percentage points with an LSTM/SATRN decoder, respectively, while saving 12 million parameters and 5.5 billion FLOPs in either pairing. Replacing the LSTM decoder with the SATRN decoder adds 11 million parameters and 19.5 billion FLOPs. Overall, full SATRN scores 89.2% versus SAR's 87.9%, but its FLOPs rise from 21.9B to 35.9B.

Multi-line Text Recognition

To evaluate the model under different text distortions, the authors ran separate tests on rotated and multi-line text derived from IC13 images.

For rotated text, they trained the model using rotations distributed uniformly from 0° to 360°, adjusting the input image size to 64×64. For multi-line text, the model was trained using synthetic text datasets (SynthText and MJSynth), and manually cropped multi-line text images from IC13 were used for testing.

At the four tested rotations, 0°/90°/180°/270°, SATRN outperforms FAN and SAR in Table 4. On multi-line text it scores 63.8%, versus SAR's 46.7%. These are separate controlled tests, not the seven standard benchmarks in Table 1. In the visualized example, decoder attention jumps from one line to the next.

The image below shows attention maps from the SATRN decoder as it extracts 2D features:

Ablation Study

  • Adaptive 2D Positional Encoding (A2DPE): In SATRN-small, no positional encoding scores 83.8%, flattened 1D and concatenated 2D encodings each score 85.8%, and A2DPE scores 86.5%. A2DPE adjusts the contributions of vertical and horizontal encodings from image features; it does not change the image aspect ratio.
  • Feature-map height: With width downsampling held at 1/4, height factors 1/4→1/8→1/16→1/32 yield 86.5→85.5→83.6→81.9% accuracy while FLOPs also fall. The accuracy loss grows beyond 1/8, showing a cost–accuracy trade-off for retaining 2D height information.

Conclusion

SATRN demonstrates that Transformers can recognize irregularly arranged text using 2D features. Table 1 reports strong results on the three irregular-text datasets, but its prior methods do not all share training data, and the paper gives conflicting average-gain figures. The per-dataset results are the more precise account.

Although SATRN improves accuracy, the decoder adds substantial parameters and FLOPs compared with the LSTM decoder. The complete model uses 35.9B FLOPs versus SAR's 21.9B in Table 2. But apparently that is not a problem!

Future research can explore ways to reduce the computational load of the model for more practical applications.