Back to paper library
arXiv 2023IEEE TIP 2024Text recognitionPaper notes

CLIP4STR

The Blessing of Multimodality

Original paper

CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model

Paper authors

Shuai Zhao · Ruijie Quan · Linchao Zhu · Yi Yang

First submitted to arXiv
arXiv:2305.14014
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{textrecognitionclip4str2023,
  title = {{CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model}},
  author = {Shuai Zhao and Ruijie Quan and Linchao Zhu and Yi Yang},
  year = {2023},
  eprint = {2305.14014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.14014}
}
On this page

Contrastive learning has swept across various fields.

Following contrastive learning, multimodal learning has become a major focus in recent years, giving rise to a new term:

  • VLM, Vision-Language Models.

In the industry, almost everyone—regardless of age or background—can confidently chant: VLM!

With all the hype, it’s only natural for researchers in the field of text recognition to join the action, right?

Defining the Problem

Scene Text Recognition (STR) faces challenges when dealing with text that is rotated, curved, blurred, or occluded. Despite decades of work, STR’s performance on these challenging cases remains less than ideal.

It’s time to shake things up.

Since CLIP was introduced in 2021, its powerful cross-modal learning capabilities have made it a hot topic. CLIP’s ability to perceive and interpret various text forms in natural images holds promise as a potential solution for STR.

Wait a second—what exactly is CLIP?

Why the sudden shift in style?

CLIP

We know you might not feel like diving into the details, so here’s a brief overview of CLIP.

Below is the architecture of CLIP:

Imagine we have a set of image-text pairs, where one pair could be a picture of a dog and the caption “a cute little dog.”

In each training batch, CLIP processes multiple such pairs. The image encoder, possibly a ResNet or ViT, processes the images to obtain image features, while the text encoder, typically a Transformer, processes the text to obtain text features.

The model then compares these features to ensure that the cosine similarity between correctly matched images and text (e.g., a dog image and “a cute little dog”) is maximized, while the cosine similarity between mismatched pairs (e.g., a dog image and the text “an apple”) is minimized.

It’s a straightforward architecture!

Finally, stack 400 million image-text pairs, and start training!

So, What Does This Mean?

What’s the significance here?

A trained CLIP model can retrieve images from text, or rank supplied text candidates for an image. It does not generate captions by itself. Image–text pretraining may also associate written text with its visual appearance, which motivates adapting CLIP to STR.

  • Is there a link between text images and the text itself? That’s precisely what STR aims to resolve!

CLIP was trained on natural images from many settings, so its features may capture forms of text that ordinary STR training data misses. That makes it a promising starting point for improving STR.

Most notably, for distorted, skewed, or heavily occluded text, CLIP’s data includes “unimaginable” varieties, so it may be able to detect these text features.

Zero-Shot Testing

To validate this concept, the authors conducted zero-shot testing on CLIP.

They wanted to see how much text CLIP could perceive. In the image below, the left column shows the input images, the middle column visualizes attention, and the right column shows scores for the supplied class candidates:

To our surprise, CLIP responds strongly to the text sticker, even when it is rotated or curved. These are scores over the supplied class candidates, though, rather than unrestricted transcription.

For the occluded example, both “cat” and “horse” receive weight, suggesting sensitivity to the visible text and image context. The displayed softmax scores depend on the candidate set and are not calibrated probabilities over all possible words.

Problem Solving

Since CLIP already has a certain degree of text understanding, our goal now is to "guide" it, focusing its capabilities on the STR task.

Model Architecture

Assuming you’re already familiar with CLIP, let’s dive straight into this architecture.


CLIP has two branches: a text branch and an image branch.

In the CLIP4STR architecture, the image is first fed into the "CLIP image branch" to obtain image features. These features are then passed into a custom “image decoder” to process the features and generate an initial text prediction.

As shown above, when an image is input, the model might first output something like "briiad." Since this output may be incorrect, it is then fed into the "CLIP text branch" to extract text features.

Finally, the text and image features are concatenated and fed into a custom "cross-modal decoder" to refine and produce the final text prediction. At inference, both branches first make autoregressive predictions in sequence; cloze masking can then refine them iteratively, with the cross-modal output used at the end.

The default configuration freezes the first half of the text encoder, rather than the entire text branch. The image encoder is fine-tuned through the visual branch. A stop-gradient on image features supplied to the cross-modal branch blocks that branch's backward path, allowing the visual branch to learn independently.

The CLIP text branch supplies a representation of the provisional text for correction; it does not generate the initial prediction.

Text Decoder

Remember the "cross-modal decoder" module we mentioned earlier?

This module follows PARSeq's permuted sequence modeling and Transformer decoder design. The visual and cross-modal decoders have the same structure, but their second attention layer reads different features: the visual decoder uses image features, while the cross-modal decoder uses token-level features concatenated from the CLIP text and image encoders.

Discussion

Comparison with Other Methods

The imported table compares nine STR benchmark settings and separates synthetic from real training data. CLIP4STR reaches high accuracy in several columns, but the training data and model size differ between rows and must be considered when comparing methods.

Basic Ablation Study

The study begins with a baseline model that only includes a ViT-S encoder for the visual branch, with no pre-training.

  1. Introducing PSM (Permuted Sequence Modeling) and following PARSeq’s training recipe brings a 0.7-percentage-point accuracy improvement.
  2. Replacing the encoder with CLIP’s image encoder ViT-B/16 changes average accuracy only from 89.9% to 90.0%, indicating a need for further adaptation.
  3. Adjusting training parameters by setting the patch size to 16×16, using a smaller learning rate for the encoder, a larger one for the decoder, and reducing training epochs to 16 rounds.

At this point, the model has already surpassed previous SOTA performance.

  1. Adding the cross-modal branch with features from all CLIP tokens raises the nine-benchmark average from 90.8% to 91.2%; using only [CLASS] and [EOS] gives 90.6% (added in the paper's v4 Table V; the figure above does not show this row).
  2. Introducing a larger model, ViT-L/14, yields an additional 0.7-percentage-point improvement in accuracy.

The CLIP-ViT-L/14 converges faster on STR than CLIP-ViT-B/16, requiring only 10 training epochs.

How Should We Freeze CLIP?

#Params counts trainable encoder parameters, while token indicates the use of pre-trained token embeddings only.
The top half shows the frozen text branch, and the bottom half shows the frozen image branch.


Common freezing strategies include:

  1. Freezing part of the CLIP text branch: The default freezes 6 of its 12 layers, and the paper tests other frozen-layer counts.
  2. Freezing the CLIP image branch: This makes the image branch untrainable, which could impact final performance.

With different numbers of frozen text-encoder layers, results rise or fall across the five tested benchmarks. Freezing many image-encoder layers causes a clearer decline.

A variant using only the pretrained text token embeddings also comes close to the default setting. The authors take this as evidence that the language information needed for STR is often about words and phrases rather than the full range of language understanding tasks.

Freezing the image model has a larger impact, possibly because CLIP was pretrained mainly on natural images while STR uses cropped word images. The results support keeping the image encoder trainable to narrow this gap.

Is Multimodality Really Useful?

To verify whether multimodal pre-training truly benefits STR, the authors re-trained the “image branch.”

Three different image encoders were tested:

  • Randomly Initialized ViT: No pre-training.
  • ImageNet-1K Pre-trained ViT: Pre-trained on ImageNet-1K.
  • ImageNet-21K Pre-trained ViT: Pre-trained on ImageNet-21K.

This table compares only CLIP4STR-B's visual branch. Image-text pretraining outperforms training from scratch on four of the five benchmarks and ties it at 86.6% on Uber. Both ImageNet-1K and ImageNet-21K pretraining fall below the scratch result in all five columns.

Can We Use PEFT?

Aside from full fine-tuning, parameter-efficient fine-tuning (PEFT) methods for large pre-trained models are becoming increasingly popular. For example, CoOp trains only learnable prefix prompts, while CLIP-Adapter adds tunable linear layers on top of a frozen VLM.

Given the success of PEFT methods in some tasks, the authors applied two PEFT methods to STR in this study:

  1. CLIP-Adapter: Adds two linear layers on top of the frozen CLIP model, with a residual addition ratio of λ=0.2\lambda = 0.2.
  2. LST (Ladder Side-Tuning): Uses a ladder side network with the frozen CLIP model, where features are downsampled, then upsampled to match the original feature dimensions.

For CLIP4STR-B's visual branch, CLIP-Adapter performs better than leaving the model frozen, but worse than full fine-tuning. LST improves accuracy further, though it also remains below full fine-tuning.

When training resources are limited, LST serves as a viable alternative.

Conclusion

Multimodal models have done well across many fields in recent years. This study shows how CLIP can be applied to STR and reports leading results in several benchmark settings.

The road has not ended; perhaps it is time to turn.