Back to paper library
IEEE TPAMI 2017Text recognitionPaper notes

CRNN

I Want It All!

Original paper

An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition

Paper authors

Baoguang Shi · Xiang Bai · Cong Yao

Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@article{textrecognitioncrnn2017,
  title = {{An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition}},
  author = {Baoguang Shi and Xiang Bai and Cong Yao},
  year = {2017},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  url = {https://doi.org/10.1109/tpami.2016.2646371},
  doi = {10.1109/tpami.2016.2646371}
}
On this page

OCR (Optical Character Recognition) is a long-established field, but deep learning has revitalized it.

Problem Definition

With the rise of deep learning, OCR gained new momentum. Compared with traditional classification methods based on handcrafted features, CNN-based OCR showed stronger performance and generalization.

One important advantage is that CNNs can learn features from images without having to design and select them by hand. This saves substantial human effort and can reduce the resources spent on a separate feature-engineering pipeline.

But a key problem in text recognition remains: How do we classify the final text effectively?

For the final step of text classification, several mainstream methods exist. Let’s borrow an image from the CHAR paper to explore these methods:

1. Dictionary Encoding

Dictionary Encoding is one of the simplest and most straightforward methods. In this approach, a dictionary is defined that contains all possible labels (usually words or phrases). When processing an image, the recognized text is classified into one of the dictionary entries. In simple terms, the model’s task is to choose the most suitable word from a pre-defined list.

However, if the text in the image is not present in the dictionary, this method fails, and the model cannot recognize out-of-vocabulary content. This limitation becomes problematic when dealing with random character strings (such as randomly generated passwords or phone numbers).

Moreover, if the dictionary is very large (e.g., hundreds of thousands of entries), the system’s efficiency is significantly impacted, and scalability becomes a challenge.

2. Character Sequence Encoding

Character Sequence Encoding is another common method, and unlike dictionary encoding, it doesn’t rely on a pre-defined dictionary. Instead, the text is directly classified into a sequence of characters. This means that the system doesn’t need to know specific words; rather, it classifies each character based on the image and constructs the complete sequence.

Character sequence encoding is more challenging because the model needs to learn the features of each character and be able to accurately combine the characters into a complete sequence. Since there are dependencies between characters in a sequence, the model requires strong contextual understanding, or it may generate incorrect character combinations.

3. N-gram Encoding

Bag-of-N-gram Encoding is a compromise between word-level and character-level methods: it represents image text through the character N-grams it contains. As a bag, it does not preserve their complete order.

An N-gram is a sequence of N characters, where N could be 2 (bi-gram), 3 (tri-gram), or even longer. Recognizing these short character groups gives the model more local context than treating characters entirely in isolation.

However, as N increases, the number of possible N-grams grows rapidly, increasing the number of classes and the computational cost. For some words, short N-grams may still miss the information carried by the complete word, so recognition can be less accurate than with other encodings.


In summary, the authors aim for a model that meets several demands:

  1. End-to-End Training: The convolutional, recurrent, and transcription components are optimized jointly.
  2. Convolutional Features: Use convolutional features learned from training images, rather than handcrafted ones.
  3. No Need for Character-level Annotations: The model learns text sequences directly from images without character-level annotations.
  4. Unrestricted: Able to recognize sequences outside a fixed word lexicon, within the character set used for training.
  5. Compact Model: Small storage size with high efficiency.

This might seem ambitious, but let’s see how the authors approached the problem.

Solution

Model Architecture

The CRNN model is composed of three parts, illustrated in the diagram:

Convolutional Layers

Starting from the bottom of the diagram, this part represents the convolutional neural network (CNN) used to extract features from the image.

Suppose the input text image is 128x32 (width × height). The sequence model receives the columns of the CNN output feature map, not the 128 pixel columns of the input image; the features in each output column are concatenated into a vector.

One potential approach is to send the raw image directly into a sequence model.

However, this could make the model overly complex, and training becomes difficult when sequence lengths are too long. Therefore, the authors first extract features from the image using a convolutional network, and these features are then sent into the sequence model.

But using a convolutional network raises another issue: if it downsamples too much horizontally, the resulting sequence will be too short.

In our example, if the input width of 128 is repeatedly halved with stride 2, few feature columns will remain.

This is far from ideal since there isn’t enough information to make predictions!

So the authors change the 3rd and 4th MaxPooling windows to rectangular 1×2 (width × height), preserving more feature columns along the horizontal axis.

The whole CNN still reduces width: in the paper's example, a 100×32 image yields a feature sequence of about 25 time steps.

Recurrent Layers

Next is the middle Recurrent Layers section.

Once the input text image is converted into sequential data, the next step is to feed this data into a sequence model. The authors use a Bidirectional LSTM (BiLSTM) for this purpose.

LSTM (Long Short-Term Memory) is a variation of RNN (Recurrent Neural Networks) designed to address the issue of long-term dependencies. As time steps increase, RNNs struggle to retain earlier information, especially for long-range dependencies in sequences. LSTM overcomes this by using memory cells and gating mechanisms (such as input, forget, and output gates) to control the flow of information, allowing it to retain important information over extended time periods.

BiLSTM combines an LSTM that reads from the start of the sequence with one that reads from the end, so each time step can use context from both directions. The CRNN stacks two such bidirectional layers, as shown in the figure. For tasks such as language modeling, speech recognition, and machine translation, this broader context can help prediction.

Returning to the problem at hand, after extracting features from the image using a CNN, the sequence length is the width of the CNN output feature map, not the original image width. Each time step can be seen as representing a block of the original image (as shown in the image below), and the receptive field of each time step depends on the design of the convolutional network.

By feeding this feature sequence into the BiLSTM, we obtain a higher-level representation of the sequence, which can then be used for predicting the text in the image.

Unaligned Sequence Labeling

This is the Transcription Layer in the diagram above.

As we discussed earlier, character-level labeling is a labor-intensive task.

To address the issue of aligning characters to sequences, the authors introduce the CTC (Connectionist Temporal Classification) technique for predicting text sequences.

CTC is designed to handle unaligned sequence labeling problems. Given frame-wise predictions y={y1,...,yT}y = \{y_1, ..., y_T\}, it assigns a probability to a label sequence ll without requiring the position of each label. This makes it particularly suitable for tasks like speech recognition and handwriting recognition.

Unlike traditional sequence models, CTC doesn’t require an exact match between each time step and character position, simplifying the training process.

Given the input sequence y={y1,...,yT}y = \{y_1, ..., y_T\}, where TT is the sequence length, each yt∈R∣L′∣y_t \in \mathbb{R}^{|L'|} represents a probability distribution over the label set L′=L∪{blank}L' = L \cup \{\text{blank}\}, where LL includes all task-related labels (such as alphabet letters), and "blank" represents an empty label. The blank label is used to denote time steps where no character is produced, which helps handle irregular sequence lengths.

CTC uses a mapping function BB that first merges adjacent repeated labels, then removes blanks. Repeated characters separated by a blank remain distinct; reversing these operations would produce the wrong result.

For instance, if the model’s frame-wise output for the word "hello" is:

Code
--hh-e-l-ll-oo--

Here, - represents the blank label. Adjacent repeated labels are collapsed first, and the blanks are then removed, yielding the final sequence:

Code
hello

CTC excels at handling such redundancy and maps the output sequence to a more concise label sequence.

In CTC, to calculate the conditional probability of a label sequence ll given the frame-wise predictions y={y1,...,yT}y = \{y_1, ..., y_T\}, we sum the probabilities of all possible frame sequences π\pi that can be mapped to the same label sequence ll. This is expressed as:

p(l∣y)=∑π:B(π)=lp(π∣y),p(l | y) = \sum_{\pi : B(\pi) = l} p(\pi | y),

where the probability of each frame sequence π\pi is defined as:

p(π∣y)=∏t=1Tyπtt,p(\pi | y) = \prod_{t=1}^{T} y^t_{\pi_t},

This represents the probability of outputting label πt\pi_t at each time step tt.

Training Strategy

The authors used the synthetic dataset released by Jaderberg et al. for training on scene text recognition:

  • Text Recognition Data The dataset contains 8 million training images along with corresponding annotated text, generated by a synthetic engine with high realism.

The model was trained solely on this synthetic data, and it was directly tested on all real-world test datasets without any fine-tuning.

Details of the network configuration are as follows:

  • The convolutional layers are based on a modified VGG architecture to accommodate English text recognition.
  • In the 3rd and 4th max-pooling layers, the authors used rectangular pooling windows with w=1,h=2w=1, h=2 instead of the traditional square ones.
  • Two batch normalization layers were inserted after the 5th and 6th convolutional layers, significantly accelerating the training process.
  • The model was trained using the ADADELTA algorithm, with ρ\rho set to 0.9. All images were resized to 100×32100 \times 32 to speed up training.
  • During testing, images were scaled to a height of 32, and the width was scaled proportionally while maintaining a minimum of 100 pixels.

Evaluation Metrics

The authors evaluated the model's performance using four common scene text recognition benchmark datasets:

  1. ICDAR 2003 (IC03)

    • The test set contains 251 scene images with labeled text bounding boxes.
    • For fair comparison with previous works, text images containing non-alphanumeric characters or less than three characters are usually ignored. After filtering, 860 cropped text images remain for testing.
    • Each test image is accompanied by a 50-word lexicon (dictionary). Additionally, a full lexicon is provided, which merges the lexicons of all images for evaluation.
  2. ICDAR 2013 (IC13)

    • The test set inherits and corrects part of the IC03 data, resulting in 1,015 cropped text images with accurate annotations.
    • Unlike IC03, IC13 does not provide a lexicon, so evaluations are done without dictionary assistance (i.e., in a no-dictionary setting).
  3. IIIT 5K-Word (IIIT5k)

    • The test set contains 3,000 cropped text images collected from the web, covering varied fonts and backgrounds.
    • Each image comes with two lexicons: a small dictionary containing 50 words and a large dictionary containing 1,000 words for dictionary-assisted evaluation.
  4. Street View Text (SVT)

    • The test set comprises 249 scene images from Google Street View, cropped into 647 text images.
    • Each text image is accompanied by a 50-word lexicon for dictionary-assisted evaluation.

Discussion

Comprehensive Model Comparison

To present a more comprehensive comparison of CRNN against other methods, the authors provide the following table:

  • E2E Train: Can the feature extractor, sequence model, and transcription component be trained jointly?
  • Conv Ftrs: Does the model use convolutional features learned from training images rather than handcrafted features?
  • CharGT-Free: Can the model be trained without character-position annotations?
  • Unconstrained: Is the model restricted by a specific dictionary, or can it handle out-of-vocabulary words and random sequences?
  • Model Size: The number of model parameters in the table, rather than the file size.

From the table, it’s clear that CRNN holds advantages in several areas, such as supporting end-to-end training, eliminating the need for character-level annotations, handling arbitrary sequences without being restricted by a dictionary, and being a compact model.

Comparison with Previous Methods

The above table shows the recognition accuracy of the CRNN model on four public datasets, compared to the latest deep learning models at the time.

With a constrained lexicon, CRNN outperforms most methods in the table. Compared with Jaderberg et al. [22], it scores higher on the lexicon-assisted IIIT5k and SVT settings, but lower on IC03 with the Full lexicon.

Because its architecture does not depend on a predefined word lexicon, CRNN can in principle handle random strings, such as phone numbers, sentences, or other scripts such as Chinese. This table still reports English scene-text recognition; another script would require the corresponding labels and training data.

Without a lexicon, CRNN obtains the best score in the table on SVT, while some methods score higher on IC03 and IC13. The starred Jaderberg et al. [22] entries are still constrained by a 90k-word dictionary.

Broader Applications

Who says OCR is only for text?

CRNN can be applied beyond text recognition to other domains, such as Optical Music Recognition (OMR).

Traditional OMR methods typically require image preprocessing (e.g., binarization), staff line detection, and individual note recognition. The authors reframed OMR as a sequence recognition problem, using CRNN to predict note sequences directly from images.

  • For simplicity, the task here focuses on pitch recognition, ignoring chords, and assumes all sheet music is in the key of C major.

To prepare training data for CRNN, the authors collected 2,650 images from the Musescore website:

Each image contains 3 to 20-note fragments, with manual annotations of the pitch sequences. Data augmentation techniques like rotation, scaling, and noise addition expanded the training samples to 265k images.

To evaluate the model, the authors compared it to two commercial OMR engines: Capella Scan and PhotoScore.

This OMR experiment uses a smaller CRNN: the 4th and 6th convolutional layers are removed, and bidirectional LSTM is replaced by a unidirectional LSTM. Each table cell reports fragment accuracy / average edit distance. CRNN has higher accuracy and lower edit distance on all three sets, for this restricted pitch-recognition task.

Capella Scan and PhotoScore perform relatively well on the "Clean" dataset but struggle on synthetic and real-world data.

This is mainly because these systems rely on robust binarization for detecting staff lines and notes. However, under suboptimal conditions, such as poor lighting, noise, or cluttered backgrounds, the binarization step often fails. In contrast, CRNN’s convolutional features are highly resilient to noise and distortion.

Additionally, CRNN’s recurrent layers can leverage the contextual information in sheet music. The recognition of each note is influenced not only by the note itself but also by its surrounding notes. For example, by comparing the vertical position of notes, certain notes can be recognized more accurately.

Conclusion

Let’s revisit the authors’ initial goals:

  1. End-to-End Training: The convolutional, recurrent, and transcription components are optimized jointly.
  2. Convolutional Features: The model uses convolutional features learned from training images rather than handcrafted features.
  3. No Need for Character-level Annotations: The model learns text sequences directly from images without character-level annotations.
  4. Unrestricted: The model can recognize sequences outside a fixed word lexicon, within its trained character set.
  5. Compact Model: Small storage size with high efficiency.

CRNN achieves all these goals. This paper is a classic work in the field, and anyone working on OCR should give it a read.