Turning a CLIP Model into a Scene Text Detector
- Publication
- CVPR 2023Conference
- First submitted to arXiv
- arXiv:2302.14338
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{textdetectiontcm2023,
title = {{Turning a CLIP Model into a Scene Text Detector}},
author = {Wenwen Yu and Yuliang Liu and Wei Hua and Deqiang Jiang and Bo Ren and Xiang Bai},
year = {2023},
eprint = {2302.14338},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2302.14338}
}
Contrastive learning has left a deep impression in many fields.
Naturally, researchers in text detection also want to join the wave.
Defining the Problem
Multimodal applications are very popular, especially after the release of the CLIP model. Its simple concept has attracted researchers like a magnet.
Everyone is trying to apply pre-trained models to their own fields in order to ride the latest trend.
Wait! You said you still don’t know what CLIP is?
CLIP
This chapter has been repeated several times, so let's briefly introduce CLIP again.
The diagram below shows the CLIP architecture:
Suppose we have a set of image-text pairs, one of which might be an image of a dog and the text "a cute little dog."
In a training batch, CLIP will receive multiple such pairs. The image encoder might use ResNet or ViT to process the image and extract features, while the text encoder might use Transformer to process the text and extract features.
The contrastive objective raises the relative similarity of matched image–text pairs and lowers that of unmatched pairs in the batch. It encourages this separation without guaranteeing perfect separation of every example.
Finally, the model is trained using 400 million image-text pairs.
Multimodal Neurons
Shortly after CLIP was released, OpenAI conducted some research on the CLIP model and discovered some interesting phenomena.
In this paper, it was pointed out that CLIP's neurons have specialized functions, such as neurons responsible for processing emotions or those responsible for geographic locations.
One of the figures related to text is shown below. Let’s take a look:
The far-left image is the original image input, and the model doesn’t show any errors; it correctly recognizes the content of the image.
But from the second image onward, things get interesting.
In the subsequent inputs, text interference is overlaid, such as a block of text saying "ipad" or "pizza." We were surprised to discover that:
- The model gets misled!
In these examples, overlaid text misleads CLIP’s image classification.
So What Should We Do?
OpenAI researchers considered this a serious issue because it means the CLIP model is too sensitive to text. In real-world applications, it could be vulnerable to attacks.
Malicious behavior can manipulate the model's recognition results through text interference, leading to incorrect (but advantageous to the attacker) outcomes. Therefore, for OpenAI, the challenge is to overcome text attacks and prevent text from easily influencing the final image recognition results.
But for text detection researchers:
- This is not a bug, this is a feature!
It turns out that CLIP is a model inherently friendly to text, which is great news for both text detection and recognition fields!
So, not only should we use it, but we should use it to the fullest!
Solving the Problem
To effectively leverage the pre-trained knowledge from CLIP, there have been many studies in the past literature, with CoOp being one of the more famous examples.
In simple terms, the image is fed into the image branch of CLIP to extract image features; then, the prompt text is fed into the text branch to extract text features. This method transfers CLIP's knowledge and applies it to text detection tasks.
Text prompts are one route for transferring knowledge. TCM also fine-tunes the image encoder at a lower learning rate and uses image-conditioned prompts to connect the visual and text features.
In the CoCoOp study referenced earlier, researchers used image features to generate prompts for the text branch, enhancing its performance through adaptive prompt learning. This technique has been widely applied in subsequent studies.
In this article, the authors also propose a model architecture based on this technique to extract knowledge from CLIP.
Model Architecture
The model architecture is named TCM by the authors, which might seem a bit complex, but let's break it down step by step:
First, the authors use CLIP's pre-trained ResNet50 as the image encoder to map the input image to the embedding vector , where:
The formula is represented as follows:
Language Prompt Generator
This is the Language Prompt Generator in the diagram above.
In the previous step, we obtained the image features.
Next, we need to activate the knowledge in the text branch. Based on CoOp’s inspiration, we add learnable prompts to the input of the text encoder to enhance the transfer ability of the text embeddings. The formula is as follows:
The language prompt generator uses the image features obtained earlier to generate the prompt , as shown in the following formula:
Where .
Text Encoder
The frozen CLIP pre-trained text encoder is used in the paper, and the predefined prompt is set as "Text."
The text encoder embeds the predefined single-class prompt into the continuous vector space, as shown in the formula below:
Where .
Finally, the prompt and text embeddings are combined to get the final text embedding :
Visual Prompt Generator
This refers to the Visual Prompt part in the diagram above.
Here, a Transformer cross-attention mechanism is used to learn the visual prompt , which conveys information from the image level to the text instance level, as shown in the formula:
The image embedding is then adjusted by the prompt as follows:
After this step, the Locality Embeddings in the diagram are obtained.
Instance–Language Matching
The text embedding is matched with the adjusted image embedding through dot-product and Sigmoid activation, generating a score map for binary segmentation, with continuous values between 0 and 1:
The binary cross-entropy loss (BCE Loss) between the segmentation map and the ground truth is:
The total loss function is:
Where , and is the loss of the chosen detector. The expression above uses the negative sign required when minimizing BCE; the displayed equation in the paper omits it. The matching score map is concatenated with the adapted image embeddings before the detection head.
During the inference phase, the output of the detection head is the final result.
Training Strategy
Experiments were conducted using popular text detection methods, including DBNet (DB), PAN, and FCENet (FCE). To maintain consistency in setup, the detectors were trained on both SynthText and real datasets. Unless stated otherwise, the backbone uses the pre-trained image encoder ResNet50 from CLIP.
The visual prompt generator consists of 3 layers of Transformer decoders, utilizing 4 attention heads, with a Transformer width of 256 and a feedforward hidden layer dimension set to 1024. The detection head uses the corresponding detection heads from DBNet, PAN, and FCENet to predict the final results.
Evaluation metrics use Intersection over Union (IoU) to determine whether the model correctly detects text regions, and precision, recall, and F1 score are calculated for comparison. To ensure a fair comparison, during training and testing, regions labeled as "do not care" or "###" in all datasets are ignored.
Discussion
Integration with Existing Methods
The authors initially define TCM as a modular model that can be integrated with existing text detection methods. The table above reports the results of combining TCM with three text detection methods on the IC15, TD, and CTW datasets:
- F1 Score Improvement: TCM outperforms the original FCENet, PAN, and DBNet on the IC15 dataset by 0.9, 1.7, and 1.9 percentage points, respectively. Similar stable performance improvements were observed on the TD and CTW datasets.
- Inference Speed: TCM-PAN, TCM-FCENet, and TCM-DBNet run at 18, 8.4, and 10 FPS, respectively, versus 36, 11.5, and 14.5 FPS for their original detectors. The table reports speed by model, measured with a ResNet50 backbone on one V100; these FPS figures are not separate IC15, TD, and CTW measurements.
Pretrained CLIP Backbone
The first test involved replacing the original backbone network of DBNet with CLIP's pre-trained image encoder, ResNet50, to quantify the impact of the backbone network on performance. In the table above, CR50 refers to using the CLIP image branch, while R50 refers to using the original DBNet backbone network.
Replacing DBNet's backbone with CLIP-R50 raises the F-measure by 0.4 percentage points on IC15 and 1.9 points on TD, while TT and CTW are unchanged. The backbone swap alone does not bring a consistent gain or fully use CLIP's vision-language knowledge.
Effectiveness of Model Components
Let's first understand the abbreviations in the table above:
- BSL: DBNet baseline with the CLIP ResNet50 backbone in this table.
- PP: Predefined Prompt.
- LP: Learnable Prompt.
- LG: Language Prompt Generator.
- VG: Visual Prompt Generator.
- TCM: Full Model Configuration.
When using predefined prompts (second row in the table), performance slightly improved across the four datasets (IC15, TD, TT, CTW), with improvements of 0.05, 0.2, 0.04, and 0.1 percentage points, respectively, compared to the baseline method.
When combining learnable prompts with predefined prompts (third row), learnable prompts consistently lead to performance improvements.
Compared to using 4 prompts, when the number was increased to 32, performance significantly improved across datasets. This could be because more learnable prompts better guide the pre-trained text encoder's knowledge, aiding text detection. To simplify later experiments, the default number of learnable prompts was set to 4.
The authors also evaluated the performance of the language prompt generator (seventh row in the table).
With the help of the language prompt generator, performance improved further across the four datasets, particularly on ICDAR2015. These same-dataset ablations show that its conditional cues help detection here; the cross-dataset transfer experiments below address generalization.
When combining the visual prompt generator with the other components, F-measure was higher across all four datasets than the baseline. For the full TCM configuration with 32 learnable prompts, the gains on IC15 and TD were 1.7 and 2.0 percentage points, respectively.
This clear complementary effect arises because the visual prompt generator can propagate fine-grained visual semantic information from the text features.
Additionally, the local image embeddings generated by the visual prompt generator help guide the model to obtain more accurate text instance-level visual representations, enhancing the instance-language matching capability and producing precise segmentation maps that aid the downstream detection head.
Ablation Experiment on VG and LG's Generalization Performance
Removing the visual prompt generator (VG) and language prompt generator (LG) from TCM results in a significant drop in generalization performance, further proving the effectiveness of VG and LG.
Frozen Encoder Experiment
TCM freezes the text encoder and fine-tunes the image branch to retain language priors. This is a design choice for TCM, not a requirement for every use of CLIP. The authors vary the learning-rate factors of both encoders to study this choice.
Among the four settings in the table, an image-encoder learning-rate factor of 0.1 with the text encoder frozen (factor 0.0) gives the highest TD500 F-measure, 88.7%.
Increasing Training Data
To further investigate whether TCM can learn additional knowledge that is difficult to acquire just by adding more data, the authors conducted the following experiment:
- Training Data: The model was trained using large-scale public datasets, including IC13, IC15, TD, CTW, TT, and MLT17, with a total of 13,784 images.
- Test Data: The model was tested on night-time data carefully collected from ArT (326 images). A sample of the night-time data is shown in the image below.
The experimental results are shown in the table below:
Even with the addition of a large amount of training data, existing methods still show limitations on night-time images that differ from the training distribution. In this NightTime-ArT test, TCM-DBNet reaches 70.2% F-measure, above DBNet's 52.8% and FCENet's 55.2%.
Increasing Parameter Count
For a fair comparison, the authors increased the parameter count of DBNet by replacing the backbone with a larger ResNet and conducted experiments on the TD500 dataset:
TCM-DBNet uses 50M parameters and 156G FLOPs, between DBNet-R101 and DBNet-R152. Its 88.7% F-measure exceeds DBNet-R152's 87.3% despite the latter's 62M parameters and 180G FLOPs.
Impact of Auxiliary Loss
Finally, the authors compared the results with and without using auxiliary loss on the TD500 dataset:
With auxiliary loss, TCM-DBNet reaches 88.7% F-measure on TD500, versus 85.1% without it. The auxiliary loss constrains the instance-language matching score map and helps train the model.
Cross-Dataset Generalization
The authors trained on IC13 and tested transfer to IC15 and TD. TCM-DBNet reaches 71.9% and 65.1% F-measure, respectively, above the DBNet baseline's 63.9% and 53.8% in the same table.
Conclusion
Everyone wants to use CLIP to enhance their models, and this paper serves as an excellent example.
TCM adapts knowledge from an already pretrained CLIP model to scene text detection without an additional text-specific pretraining task.
Although the performance improvement of TCM in the current experiments is not very significant, the cross-dataset generalization experiments show that pre-trained multimodal models can provide broader knowledge, which helps improve the model's generalization ability.
Overall, TCM offers a valuable direction and idea for future research.