Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Publication
- NeurIPS 2021Conference
- First submitted to arXiv
- arXiv:2107.07651
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityalbef2021,
title = {{Align before Fuse: Vision and Language Representation Learning with Momentum Distillation}},
author = {Junnan Li and Ramprasaath R. Selvaraju and Akhilesh Deepak Gotmare and Shafiq Joty and Caiming Xiong and Steven Hoi},
year = {2021},
eprint = {2107.07651},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2107.07651}
}
To excel in VQA (Visual Question Answering), using an object detector is almost inevitable.
Those pioneers who attempted to do without object detectors, like ViLT and MDETR, have already faced significant challenges.
ViLT replaced the object detector with ViT but did not achieve remarkable results on the VQA v2 dataset. MDETR switched from Faster RCNN to DETR but didn't provide performance results on the VQA v2 dataset either.
On the other hand, the popular CLIP model:
These methods use contrastive loss for pretraining, focusing on learning separate representations for images and text. However, such methods may lack the modeling capability for more complex V+L tasks.
So, what are the current issues with existing approaches?
Problem Definition
The authors of ALBEF identified several key issues based on previous research:
-
Modality Space Encoding
Even when image features and text encodings are converted into vector forms, they usually exist in their unique vector spaces. These spaces have distinct dimensions, ranges, and characteristics, making their interactions and relationships challenging to capture directly.
-
Object Detector Limitations
Object detectors require bounding box annotations during pretraining, increasing annotation costs. They also require high-resolution images (e.g., 600×1000) during inference, adding computational cost and complexity.
-
Data Noise Issue
Widely used image-text datasets often come from the internet, which inherently carries noise due to diverse sources. When pretraining models with these datasets, there's a risk that the model might overfit to specific noisy text, reducing its generalization performance in real-world applications.
Summarizing these issues, the authors proposed ALBEF, achieving state-of-the-art results in both retrieval and inference tasks without relying on object detectors by using ViT. One of the reasons for this success is the inclusion of "knowledge distillation."
Knowledge Distillation
Before discussing how the authors solved the problems, let's briefly introduce "knowledge distillation."
The next few examples describe distillation in general. ALBEF itself uses online self-distillation: its teacher is an exponential-moving-average copy of the student, not a separately pretrained, larger model.
With the success of deep learning models in various tasks, model size and complexity have increased. However, large models require significant computational resources, which may not be feasible for practical applications, especially on edge or mobile devices.
Knowledge distillation is a technique in deep learning aimed at transferring knowledge from a complex (and usually large) model (the "teacher model") to a simpler (and usually smaller) model (the "student model").
Basic Principle
-
Soft Targets: In the distillation process, the outputs of the teacher model (typically the output of an activation function or classification probabilities) are used as soft targets to train the student model. These soft targets contain the teacher model's knowledge and provide more information than the original "hard" labels.
-
Temperature: A parameter called "temperature" is used to soften the probability outputs, making them smoother. Higher temperatures lead to softer probability distributions, helping the student model learn.
Implementation Steps
-
Training the Teacher Model
The teacher model is usually a large, deep learning model trained to high performance on a specific task. For example, in image classification, the teacher model could be a deep convolutional neural network like ResNet-152 or VGG-19.
-
Generating Soft Targets
Once the teacher model is trained, it can produce soft targets, which means using the teacher model's predictions as targets instead of the original hard labels.
-
Training the Student Model
The student model, being smaller, is trained to mimic the teacher model's behavior using the soft targets generated in the previous step. This way, it learns to replicate the teacher model's decision process.
In practice, both soft targets and original hard labels are often combined to train the student model, ensuring it not only mimics the teacher but also considers the actual data labels. The student model is typically much smaller and simpler than the teacher model, making it faster in inference and more suitable for resource-constrained environments.
-
Extensions and Variants
-
Self-distillation: The model uses its own predictions as soft targets for retraining, reinforcing its learned knowledge.
- Advantages: Simple and low-cost; no need for a separate teacher model. Helps model convergence and improves accuracy.
- Disadvantages: Model may overfit as it relies continuously on its predictions.
-
Multi-teacher Distillation: Knowledge from multiple teacher models is distilled into one student model, combining and mimicking all teacher models' behaviors.
- Advantages: Richer knowledge sources; more robust and accurate student model. Reduces bias from individual models.
- Disadvantages: High computational cost; increased complexity in combining and weighting outputs from different teachers.
-
Online Distillation: The teacher and student models are trained simultaneously. The student model learns from the teacher model in real-time as both are updated.
- Advantages: Real-time updates; no need to wait for one model to finish training. Lower latency.
- Disadvantages: Requires careful synchronization; increased computational demands.
-
Solution
ALBEF Model Design
Ignoring the knowledge distillation part for now, the other components should be straightforward:
- Image Encoder: Uses a 12-layer Vision Transformer (ViT-B/16), initialized with ImageNet-1k pretrained weights.
- Text Encoder: Uses a 6-layer Transformer Encoder, initialized with the first 6 layers of the BERTbase model.
- Multimodal Encoder: Uses a 6-layer Transformer Encoder, initialized with the last 6 layers of the BERTbase model.
Pretraining Objectives
-
Image-Text Contrastive Learning (ITC)
This objective aims to learn better unimodal representations before fusion, focusing on learning a similarity function that scores image-text pairs higher.
Two linear projections (gv and gw) map the [CLS] embeddings to normalized low-dimensional (256-d) representations. Inspired by MoCo, the authors use two queues to store recent M image-text representations from the momentum unimodal encoders. For each image and text, softmax-normalized image-to-text and text-to-image similarities are calculated. The image-text contrastive loss is defined based on the cross-entropy (H) between (p) and (y).
-
Masked Language Modeling (MLM)
MLM is a pretraining strategy designed to enhance a deep learning model's language understanding and cross-modal learning capabilities. ALBEF selects 15% of text tokens, masks or replaces them following BERT's procedure, and predicts the original tokens from the remaining text and image. The upper-right label in the imported architecture image says “MIM,” but the paper's method and Figure 1 caption specify MLM here.
This method is more than just a fill-in-the-blank game. It forces the model to understand the text's context and the visual content of the images and their relationship with the text. When the model is trained to optimize this prediction task, it simultaneously learns how to deeply and finely understand both text and images.
-
Image-Text Matching (ITM)
ITM is a strategy primarily used to evaluate how well a model matches text descriptions with images. This task involves assessing the deep semantic connections between text and images.
For this task, a special token called [CLS] is used as input, prompting the model to generate a single representation that fuses the image and text. This fused representation provides a unified view for the model to determine if the image and text match.
To sample hard negatives for ITM, the authors reuse the contrastive similarities already computed for ITC, then draw a relatively similar but unmatched caption for each image and an unmatched image for each caption within the same mini-batch. This is separate from the momentum queue used for ITC. The paper's “zero additional computational overhead” refers to not recomputing those similarities with another encoder pass; sampling and ITM training still have costs.
Momentum Distillation (MoD)
Despite the detailed pretraining strategies, there's still an unresolved issue:
- Noise
The pretraining image-text pairs often come from the internet, inherently noisy, and the image-text relationship is not always clear. In ITC learning, text considered a negative example (i.e., not matching the image) may still be relevant to the image. In MLM, words that describe the image well but differ from the original annotation might be incorrectly considered wrong. Existing ITC and MLM using one-hot labels penalize all predictions considered negative, regardless of their accuracy.
To address the noise in pretraining image-text pairs, the authors propose a strategy called momentum distillation. This strategy's core idea is to use an evolving "teacher model" called the momentum model to generate pseudo-targets for training the "base model."
The momentum model is not static but consists of exponential moving average (EMA) versions of the unimodal and multimodal encoders, allowing the model to adapt and reflect the latest trends in training data.
For example, suppose we have a photo of a golden beach with gentle waves lapping at the shore and a few children playing in the distance. However, the text description reads, "A golden beach with gentle waves caressing the shore, children playing joyfully, and vendors selling ice cream nearby."
This is a hypothetical noise example. In MLM, if “ice cream” is the masked word in the original caption, a hard target accepts only that word—even if there is no ice cream in the photograph. That is not ideal.
With momentum distillation, the EMA teacher's probability distribution at the masked position is mixed with the hard target. If the teacher also assigns probability to other reasonable words, the student need not put all its probability on “ice cream.” The paper does not show that the teacher necessarily recognizes the true contents of this hypothetical photograph.
This gives the student a softer training target; whether it reduces the effect of noise is an empirical question.
For ITC tasks, the authors use the momentum unimodal encoders to produce bidirectional soft pseudo-targets from image-text similarities. The ITC MoD loss keeps the original hard-label cross-entropy with weight and adds KL divergence from the teacher distribution to the student prediction with weight .
For MLM, in addition to the original MLM loss, the KL divergence between the model's predicted probabilities and pseudo-targets is also considered. This strategy is extended to downstream tasks, where each task's loss is a weighted combination of the original task loss and KL divergence.
Dataset
- Internet Datasets: This study used two major internet datasets, Conceptual Captions and SBU Captions.
- In-domain Datasets: Additionally, two in-domain datasets, COCO and Visual Genome, were used.
- In total, 4.0M unique images and 5.1M image-text pairs were collected across all datasets.
- Furthermore, to verify the method's adaptability to larger and noisier internet data, the Conceptual 12M dataset was included, increasing the total number of images to 14.1M.
Technical Details
- Model Architecture: The model comprises BERTbase (with 123.7M parameters) and ViT-B/16 (with 85.8M parameters).
- Hardware Configuration: Training was conducted on 8 NVIDIA A100 GPUs.
- Training Details: A batch size of 512 was used, with a total of 30 epochs for pretraining.
- Optimizer: AdamW with a weight decay of 0.02 was used.
- Learning Rate Schedule: The learning rate was warmed up to 1e-4 over the first 1000 iterations and then decayed to 1e-5 following a cosine schedule.
- Image Processing: During pretraining, images were randomly cropped to 256×256 resolution and RandAugment was applied. For finetuning, the resolution was increased to 384×384.
- Other Parameters: The momentum parameter for updating the momentum model was set to 0.995, the queue size for image-text contrastive learning was 65,536, and the distillation weight α was linearly increased from 0 to 0.4 over the first epoch.
Discussion
Contribution of Knowledge Distillation
Since knowledge distillation is the main feature of this paper, let's first look at the relevant data.
From the results, the following points can be observed:
- Gain from Image-Text Contrastive Learning: With 4M pretraining images, adding ITC to the MLM+ITM baseline raises all five reported downstream metrics.
- Effect of Contrastive Hard Negative Mining: Adding in-batch hard negatives to ITC+MLM+ITM raises all five metrics again.
- Contribution of Momentum Distillation: Adding MoD to ITC alone nudges NLVR² down from 80.35 to 80.34, while the other four metrics rise. After MoD for MLM and downstream distillation are added, all five exceed the preceding configuration.
- Handling Noisy Internet Data: The 14M-image model improves after more web data is added, but this table does not isolate how much of that gain comes from MoD's resistance to noisy labels.
Taken together, these ablations support ITC, hard negative mining and the complete MoD configuration in the reported setting. The noise explanation still has to stay within those experimental conditions.
Comparison with CLIP
In the Flickr30K zero-shot retrieval table, ALBEF with 4M pretraining images reaches 90.5 TR@1 and 76.8 IR@1, above CLIP and ALIGN. At 14M images these rise to 94.1 and 82.8. The 99.7 TR@10 score ties ALIGN, however, and whether a still larger dataset would keep helping remains an extrapolation.
Results on Various V+L Tasks
In other V+L understanding tasks, the table above shows ALBEF's strong performance. Using 4M pretraining images, it is already competitive; at 14M it exceeds the listed methods, including object-tag-based OSCAR and adversarially augmented VILLA.
Specifically, the 14M-image ALBEF row exceeds VILLA on VQA test-std, NLVR² test-P and SNLI-VE test by 2.37, 3.84 and 1.88 percentage points. The paper also reports NLVR² inference more than 10 times faster than VILLA under its setup; the pictured table lists accuracies, not latency.
Visualization Results
The picture above is a Grad-CAM word heatmap for a COCO caption: attention falls near objects, attributes or relationships. It is not the RefCOCO+ results table. Separately, the paper's Table 5 reports weakly supervised RefCOCO+ grounding, where ALBEF_itm scores 58.46 / 65.89 / 46.25 on Val / TestA / TestB. The two variants obtain their maps differently:
- ALBEF_itc: Visualized using Grad-CAM on the self-attention map of the last layer of the image encoder, maximizing image-text similarity gradients.
- ALBEF_itm: Uses Grad-CAM on the multimodal encoder's third-layer cross-attention map; the paper notes that this layer is more specialized for grounding, and the gradient comes from the image-text matching score.
Conclusion
One concern with knowledge distillation is how much the student can learn from the teacher's imperfect targets. A better teacher can help, although its score is not a strict upper bound on the student. That broader question is not our focus here.
The authors of ALBEF clearly understand knowledge distillation, stating:
- Knowledge distillation is used to reduce the effect of noisy labels.
The hope is that the momentum teacher assigns some probability to reasonable alternatives instead of blindly enforcing a noisy one-hot target.
This is an ideal assumption. From the noise-removal perspective, knowledge distillation is a powerful strategy but comes with challenges such as defining noise, over-reliance on the original model, the black-box nature of the distillation process, and the potential risk of over-optimization.
ALBEF demonstrates outstanding results across various V+L tasks, offering efficient and effective inference, highlighting its immense potential in this field. This also provides new directions for future research, such as better utilizing knowledge distillation to improve model generalization and further narrowing the gap between teacher and student models.