Back to paper library
arXiv 2020CVPR 2020Text recognitionPaper notes

SEED

The Seed of Language

Original paper

SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition

Paper authors

Zhi Qiao · Yu Zhou · Dongbao Yang · Yucan Zhou · Weiping Wang

Publication
CVPR 2020Conference
First submitted to arXiv
arXiv:2005.10977
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionseed2020,
  title = {{SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition}},
  author = {Zhi Qiao and Yu Zhou and Dongbao Yang and Yucan Zhou and Weiping Wang},
  year = {2020},
  eprint = {2005.10977},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.10977}
}
On this page

It seems that models relying solely on visual features have hit a performance plateau.

So, is there a way to incorporate more linguistic information into the process?

Problem Definition

Inspired by advancements in NLP, many Scene Text Recognition (STR) studies have begun to adopt attention mechanisms to tackle challenges.

For regular text, encoder-decoder architectures typically use convolutional neural networks (CNNs) and recurrent neural networks (RNNs) with attention mechanisms to predict characters step by step. For irregular text, multi-directional encoding and 2D attention methods are applied.

So...

Since STR models already borrow from NLP frameworks, why not take it a step further?

The "language" aspect in language models is actually a valuable clue.

The authors introduce extra semantic information as a global reference to help the decoder. This was an early exploration of how language and visual information might work together in scene text recognition.

Solution

Model Architecture

Paper Figure 3; source: arXiv:2005.10977v1

Building on ASTER, the authors propose the "Semantics-Enhanced ASTER (SE-ASTER)" method, as illustrated above.

SE-ASTER consists of four main modules:

  1. Rectification Module: For correcting irregular text images;
  2. Encoder: For extracting rich visual features;
  3. Semantic Module: For predicting semantic information from visual features;
  4. Decoder: For producing the final recognition result.

Initially, the input image passes through the Rectification Module, where shallow CNN layers predict control points, and the image is rectified using Thin-plate Splines (TPS).

The rectified image is then fed into the Encoder, which generates visual features. The Encoder comprises a 45-layer ResNet CNN and a BiLSTM with 256 hidden units, producing a feature sequence h=(h1,…,hL)h = (h_1, \dots, h_L) with dimensions L×CL \times C, where LL represents the width of the last CNN feature map and CC is the feature dimension.

The output feature sequence hh serves two purposes:

  • Provides input for the Semantic Module to predict semantic information;
  • Serves as input to the Decoder.

To predict semantic information, the feature sequence is flattened into a one-dimensional feature vector II with dimension K=L×CK = L \times C.

Semantic information SS is then calculated using two linear layers with a nonlinear activation between them:

S=W2σ(W1I+b1)+b2S = W_2\sigma(W_1I + b_1) + b_2

where W1,W2,b1,b2W_1, W_2, b_1, b_2 are trainable weights, and σ\sigma is the ReLU activation function. The semantic information is supervised with pre-trained word embeddings from a FastText model.

Pre-trained Language Model

The authors use pretrained FastText word vectors for semantic supervision. FastText here is a skip-gram word-embedding model, not an autoregressive text generator.

Assume that T={wi−l,…,wi+l}T = \{w_{i-l}, \dots, w_{i+l}\} is a context window centered on wiw_i, where ll is the window radius. In the skip-gram model, a word wiw_i is represented by an embedding vector viv_i and is input to a simple feed-forward neural network with the objective of predicting the context:

Ci={wi−l,…,wi−1,wi+1,…,wi+l}C_i = \{w_{i-l}, \dots, w_{i-1}, w_{i+1}, \dots, w_{i+l}\}

During training, the embedding vector viv_i is optimized such that each word's final embedding vector closely matches that of semantically similar words. FastText further embeds subwords and uses them to form the final embedding of word wiw_i.

For example, with subword lengths from 2 to 4, "where" yields wh, he, er, re, whe, her, ere, wher, and here. This simplified example omits word-boundary markers, which FastText also uses when forming character n-grams. A word is represented using its subword vectors together with its own word vector.

FastText can construct a vector for an out-of-vocabulary word from its learned subwords.


During training, FastText word embeddings supervise the semantic information SS predicted from the image. At inference, predicted SS passes through a linear transformation to initialize the decoder GRU's hidden state; the ground-truth transcription is not available then. This step matters: in the ablation, word-embedding supervision alone brings no clear improvement.

Loss Function

During training, the authors apply supervision signals to both the Semantic Module and Decoder, enabling SE-ASTER to train end-to-end.

The loss function is defined as:

L=Lrec+λLsemL = L_{\text{rec}} + \lambda L_{\text{sem}}

where LrecL_{\text{rec}} is the standard cross-entropy loss, measuring the difference between the predicted probabilities and the ground truth labels.

LsemL_{\text{sem}} is the cosine embedding loss for semantic information:

Lsem=1−cos⁡(S,em)L_{\text{sem}} = 1 - \cos(S, \text{em})

where SS is the predicted semantic information, and em\text{em} is the embedding of the transcription label from the pre-trained FastText model, aiming to maximize similarity between predicted semantic information and the FastText embeddings of the target words.

λ\lambda is a balancing hyperparameter, set to 1.

Implementation Details

The pre-trained FastText model used is the official version, trained on Common Crawl and Wikipedia datasets. SE-ASTER’s recognition output contains 97 symbols, including numbers, uppercase and lowercase letters, 32 punctuation marks, end-of-sequence, padding, and unknown symbols.

Input images are resized to 64 × 256 without preserving the aspect ratio. The ADADELTA algorithm optimizes the objective function.

The recognition network has no additional pretraining or data augmentation; its FastText supervision is still pretrained. The recognition network is trained on SynthText and Synth90K datasets for 6 epochs with a batch size of 512. The initial learning rate is set to 1.0, decaying to 0.1 in the 4th epoch and 0.01 in the 5th epoch. Training is conducted on an NVIDIA M40 GPU.

For inference, image sizes match those in training. Beam search is applied in GRU decoding, retaining the top ( k = 5 ) candidates by cumulative score in all experiments.

Discussion

Comparison with Recent Methods

Paper Table 5; source: arXiv:2005.10977v1

The authors compared their proposed method with previous state-of-the-art techniques across several benchmark datasets, as shown in the table above.

Without using lexicons and with only word-level annotations, SE-ASTER achieved the top result on 2 of the 6 benchmark datasets and the second-best result on 3 others. Notably, the proposed method demonstrated excellent performance on low-quality datasets, such as IC15 and SVTP.

Against the previously published ASTER results, SE-ASTER rises from 76.1% to 80.0% on IC15, or 3.9 percentage points, and from 78.5% to 81.4% on SVTP, or 2.9 points. Against the authors' retrained ASTER baseline, the gains are 1.6 and 2.3 percentage points, respectively. Even with a weaker backbone and without character-level annotations, SE-ASTER also exceeds ScRN by 0.6 percentage points on SVTP and 1.3 points on IC15; the training conditions differ between methods.

Visualization of Results

Paper Figure 4; source: arXiv:2005.10977v1

The authors visualized results on some low-quality images, including examples with blurring or occlusion, as shown in the figure above.

In these low-quality examples, SE-ASTER corrects some of ASTER's wrong characters. The authors suggest that semantic information gives the decoder a global feature that makes it less vulnerable to image interference.

Conclusion

Bringing a language model into STR opened a new line of thought. SE-ASTER performed well on several benchmarks and showed robustness to low-quality images.

In later STR research, using language models became a major trend. Whatever side effects they might introduce, this was still a new direction worth exploring at the time.