Back to paper library
arXiv 2019NeurIPS 2019Multimodal learningPaper notes

ViLBERT

Interwoven Layers in the Beginning

Original paper

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

Paper authors

Jiasen Lu · Dhruv Batra · Devi Parikh · Stefan Lee

Publication
NeurIPS 2019Conference
First submitted to arXiv
arXiv:1908.02265
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityvilbert2019,
  title = {{ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks}},
  author = {Jiasen Lu and Dhruv Batra and Devi Parikh and Stefan Lee},
  year = {2019},
  eprint = {1908.02265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.02265}
}
On this page

Each year in academia brings numerous developments, and 2019 was especially eventful, with the wave of advancements still unfolding.

Defining the Problem

Current methods primarily train visual and language models separately before attempting to combine them.

This approach often leads to suboptimal results, as models struggle to generalize well when visual and language data are limited or biased. While models may be capable of recognizing objects in images or understanding language independently, creating meaningful connections between these two modalities remains a significant challenge.

The authors aim to depart from traditional learning approaches, designing a method to help the model simultaneously learn from both images and text, thus fostering a closer association between them.

Approach

Model Architecture

Lu et al., ViLBERT, Figure 1, arXiv:1908.02265v1

Inspired by BERT, the authors sought to develop a similar model that could learn joint representations of language and visual content from paired data.

BERT handles text. ViLBERT adds a separate visual stream so region features and words can attend to each other, rather than simply replacing text tokens with images.

As shown above, the authors propose a dual-stream architecture: one stream is dedicated to visual processing, and the other to language. These streams interact through cross-attention mechanisms, allowing variable network depth for each modality and enabling cross-modal connections at different layers.

The image input is derived from bounding boxes and visual features extracted by a pre-trained object detector, which are then combined with visual position encodings.

Pre-training Tasks

Lu et al., Figure 3, arXiv:1908.02265v1

As illustrated above, the authors propose two main pre-training tasks: masked multimodal modeling and multimodal alignment prediction.

  • Masked multimodal modeling: Predict masked text tokens; for vision, match the detector’s class distribution with KL divergence rather than reconstructing pixels or regressing region features.
  • Multimodal Alignment Prediction: This task requires the model to determine whether the image and text are aligned—whether the text accurately describes the image.

The Conceptual Captions dataset contains about 3.3 million image-caption pairs automatically collected from the web. With some image links broken at download time, the paper actually trains on about 3.1 million pairs. The captions can be noisy or short on detail, but cover varied visual content.

The experimental setup includes:

  • Language model initialization using BERT-BASE, which was pre-trained on BookCorpus and English Wikipedia.
  • Visual features derived from a pre-trained Faster R-CNN.
  • Visual-stream Transformer and co-attentional blocks with a 1,024-dimensional hidden state and 8 attention heads.
  • Training on 8 TitanX GPUs for 10 epochs.

Discussion

How Effective is ViLBERT?

Lu et al., Table 1, arXiv:1908.02265v1; dagger omits multimodal pretraining

As shown above, the ViLBERT model achieves impressive performance across various vision-language tasks.

Both dual-stream ViLBERT and the single-stream baseline are compared with and without Conceptual Captions pretraining. The dagger omits that multimodal stage while retaining the pretrained BERT and detector backbones.

Comparing ViLBERT with ViLBERT†, VQA test-dev rises from 68.93 to 70.55, RefCOCO+ val from 68.61 to 72.34, and image retrieval R@1 from 45.50 to 58.20. Those are gains of 1.62, 3.73, and 12.70 percentage points, respectively; the gain depends on the task and metric. The single-stream model also has its own before-and-after pretraining comparison in the table.

After fine-tuning, ViLBERT is competitive across all four established tasks shown. Selected VCR, RefCOCO+, and image-retrieval metrics lead the listed prior methods by roughly 7 to 11 percentage points, depending on the metric and baseline. Extending ViLBERT to these tasks is relatively straightforward: usually, the authors add a classifier and fine-tune the whole model.

Impact of Model Depth on Performance

Lu et al., Table 2, arXiv:1908.02265v1; depth counts repeated CO-TRM→TRM blocks

As shown above, the authors examine how model depth affects ViLBERT’s performance.

  • VQA and image retrieval: VQA test-dev peaks at six layers (70.55) before dipping to 70.47 at eight; fine-tuned retrieval R@1 peaks at eight (58.78). Neither rises monotonically across every depth or peaks at the same depth.
  • Zero-shot image retrieval: R@1 rises from 26.14 at two layers to 32.80 at eight, although R@5 and R@10 dip at four layers. Eight layers perform best among those tested here.
  • VCR and RefCOCO+: The best depth varies by metric. For RefCOCO+, val peaks at six layers (72.34), testA at two (78.61), and testB at four (63.14). The VCR columns also have different peaks.

Effect of Pre-training Data

Lu et al., Table 3, arXiv:1908.02265v1; compare 0/25/50/100% of one source

Across the table’s 0%, 25%, 50%, and 100% Conceptual Captions pretraining comparisons, the reported task metrics improve monotonically with the amount of data.

This ablation supports a narrower point: with the same model and data source, using more of that pretraining set helps on these tasks.

Additionally, this suggests that ViLBERT could potentially benefit from even larger datasets during pre-training, making it a promising direction for future research and applications to further optimize model performance.

Conclusion

In the early days of multimodal research, the development of dual-stream architecture marked a major breakthrough, offering a more nuanced perspective on understanding and capturing relationships between images and text.

ViLBERT can be called a milestone for this field. It tries to connect image content and words more closely, and it achieves strong results across several vision-language tasks.