ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Publication
- NeurIPS 2019Conference
- First submitted to arXiv
- arXiv:1908.02265
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityvilbert2019,
title = {{ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks}},
author = {Jiasen Lu and Dhruv Batra and Devi Parikh and Stefan Lee},
year = {2019},
eprint = {1908.02265},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1908.02265}
}
Each year in academia brings numerous developments, and 2019 was especially eventful, with the wave of advancements still unfolding.
Defining the Problem
Current methods primarily train visual and language models separately before attempting to combine them.
This approach often leads to suboptimal results, as models struggle to generalize well when visual and language data are limited or biased. While models may be capable of recognizing objects in images or understanding language independently, creating meaningful connections between these two modalities remains a significant challenge.
The authors aim to depart from traditional learning approaches, designing a method to help the model simultaneously learn from both images and text, thus fostering a closer association between them.
Approach
Model Architecture
Inspired by BERT, the authors sought to develop a similar model that could learn joint representations of language and visual content from paired data.
BERT handles text. ViLBERT adds a separate visual stream so region features and words can attend to each other, rather than simply replacing text tokens with images.
As shown above, the authors propose a dual-stream architecture: one stream is dedicated to visual processing, and the other to language. These streams interact through cross-attention mechanisms, allowing variable network depth for each modality and enabling cross-modal connections at different layers.
The image input is derived from bounding boxes and visual features extracted by a pre-trained object detector, which are then combined with visual position encodings.
Pre-training Tasks
As illustrated above, the authors propose two main pre-training tasks: masked multimodal modeling and multimodal alignment prediction.
- Masked multimodal modeling: Predict masked text tokens; for vision, match the detector’s class distribution with KL divergence rather than reconstructing pixels or regressing region features.
- Multimodal Alignment Prediction: This task requires the model to determine whether the image and text are aligned—whether the text accurately describes the image.
The Conceptual Captions dataset contains about 3.3 million image-caption pairs automatically collected from the web. With some image links broken at download time, the paper actually trains on about 3.1 million pairs. The captions can be noisy or short on detail, but cover varied visual content.
The experimental setup includes:
- Language model initialization using BERT-BASE, which was pre-trained on BookCorpus and English Wikipedia.
- Visual features derived from a pre-trained Faster R-CNN.
- Visual-stream Transformer and co-attentional blocks with a 1,024-dimensional hidden state and 8 attention heads.
- Training on 8 TitanX GPUs for 10 epochs.
Discussion
How Effective is ViLBERT?
As shown above, the ViLBERT model achieves impressive performance across various vision-language tasks.
Both dual-stream ViLBERT and the single-stream baseline are compared with and without Conceptual Captions pretraining. The dagger omits that multimodal stage while retaining the pretrained BERT and detector backbones.
Comparing ViLBERT with ViLBERT†, VQA test-dev rises from 68.93 to 70.55, RefCOCO+ val from 68.61 to 72.34, and image retrieval R@1 from 45.50 to 58.20. Those are gains of 1.62, 3.73, and 12.70 percentage points, respectively; the gain depends on the task and metric. The single-stream model also has its own before-and-after pretraining comparison in the table.
After fine-tuning, ViLBERT is competitive across all four established tasks shown. Selected VCR, RefCOCO+, and image-retrieval metrics lead the listed prior methods by roughly 7 to 11 percentage points, depending on the metric and baseline. Extending ViLBERT to these tasks is relatively straightforward: usually, the authors add a classifier and fine-tune the whole model.
Impact of Model Depth on Performance
As shown above, the authors examine how model depth affects ViLBERT’s performance.
- VQA and image retrieval: VQA test-dev peaks at six layers (70.55) before dipping to 70.47 at eight; fine-tuned retrieval R@1 peaks at eight (58.78). Neither rises monotonically across every depth or peaks at the same depth.
- Zero-shot image retrieval: R@1 rises from 26.14 at two layers to 32.80 at eight, although R@5 and R@10 dip at four layers. Eight layers perform best among those tested here.
- VCR and RefCOCO+: The best depth varies by metric. For RefCOCO+, val peaks at six layers (72.34), testA at two (78.61), and testB at four (63.14). The VCR columns also have different peaks.
Effect of Pre-training Data
Across the table’s 0%, 25%, 50%, and 100% Conceptual Captions pretraining comparisons, the reported task metrics improve monotonically with the amount of data.
This ablation supports a narrower point: with the same model and data source, using more of that pretraining set helps on these tasks.
Additionally, this suggests that ViLBERT could potentially benefit from even larger datasets during pre-training, making it a promising direction for future research and applications to further optimize model performance.
Conclusion
In the early days of multimodal research, the development of dual-stream architecture marked a major breakthrough, offering a more nuanced perspective on understanding and capturing relationships between images and text.
ViLBERT can be called a milestone for this field. It tries to connect image content and words more closely, and it achieves strong results across several vision-language tasks.