Back to paper library
arXiv 2019Multimodal learningPaper notes

VisualBERT

Gaze at the Prelude

Original paper

VisualBERT: A Simple and Performant Baseline for Vision and Language

Paper authors

Liunian Harold Li · Mark Yatskar · Da Yin · Cho-Jui Hsieh · Kai-Wei Chang

Publication
Publication venue not yet verified
First submitted to arXiv
arXiv:1908.03557
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityvisualbert2019,
  title = {{VisualBERT: A Simple and Performant Baseline for Vision and Language}},
  author = {Liunian Harold Li and Mark Yatskar and Da Yin and Cho-Jui Hsieh and Kai-Wei Chang},
  year = {2019},
  eprint = {1908.03557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.03557}
}
On this page

Around 2015, numerous cross-modal models were proposed, most of which were based on LSTM architectures.

The 2017 Transformer and later BERT-style pretraining encouraged a new wave of vision-language models. Attention was already used in earlier multimodal work such as image captioning; it did not start with the Transformer.

How could we miss out on something this exciting?

Inspired by these developments, the authors aimed to capture implicit relationships within images through the attention mechanism and believed that pre-training could effectively facilitate the learning of such relationships.

Defining the Problem

Based on previous research, the authors identified a few current challenges:

  • Unified Vision-Language Model Architecture: Many existing models are designed for specific vision-language tasks, lacking a general-purpose model that can handle a variety of tasks.
  • The Importance of Pre-training: While BERT has demonstrated the significance of pre-training, the role of pre-training in the multimodal domain has yet to be fully explored.
  • Challenges in Understanding Image Semantics: The association between images and text remains difficult to achieve effectively.

Solving the Problem

Model Architecture

The core idea of VisualBERT is to leverage the self-attention mechanism within the Transformer to automatically align elements of input text with regions within an image.

VisualBERT builds upon the BERT model and adds a set of visual embeddings, denoted as FF, to represent the image. Each f∈Ff \in F corresponds to a bounding region in the image, derived from an object detector.

Each visual embedding ff is composed of three summed embeddings:

  1. Regional Feature Embedding (fof_o): Represents the visual feature of the bounding region, computed by a convolutional neural network.
  2. Segment Embedding (fsf_s): Marks the embedding as an image embedding (in contrast to a text embedding).
  3. Position Embedding (fpf_p): When alignment information between text and bounding regions is provided in the input, this embedding represents the aligned position, set as the sum of the position embeddings corresponding to the aligned words.

The visual and text embeddings enter the same stack of Transformer layers, where self-attention lets words and regions read one another. Objective 1 above Figure 2 is masked-word prediction; Objective 2 is sentence–image matching. The drawing shows how the model is trained.

Pre-training Mechanism

VisualBERT is designed to handle both language and visual inputs. For its general pre-training stage, the authors use the COCO Karpathy training split of about 100,000 images, each with five independent captions, and initialize the model from BERT.

The training is divided into three main phases:

  1. Task-Agnostic Pre-Training: The model is first pre-trained on those COCO image–caption pairs with two visually grounded language modeling objectives:

    • Masked Language Modeling: Randomly masks some words in the text input, requiring the model to predict the masked words while leaving the vectors corresponding to image regions unmasked.
    • Sentence-Image Prediction: Since each image in the COCO dataset is associated with multiple captions, the model receives a text segment consisting of two captions. One caption describes the image, while the other has a 50% chance of being another related caption and a 50% chance of being a randomly selected caption. The model is trained to distinguish between these two scenarios.
  2. Task-Specific Pre-Training: Before fine-tuning VisualBERT for specific downstream tasks, the model undergoes additional pre-training on the task-specific dataset, with the objective of masked language modeling combined with image features. This phase helps the model adapt better to the target domain.

  3. Fine-Tuning: In the final step, the model is fine-tuned with task-specific inputs, outputs, and objectives, with training focused on maximizing performance on the target task.

Discussion

Performance on VQA

The task in Visual Question Answering (VQA) is to answer a question correctly based on a given image. Here, the VQA 2.0 dataset is used, which includes over one million questions about images from COCO.

The model is trained to predict the 3,129 most frequent answers. Image features are derived from a ResNeXt-based Faster R-CNN, pre-trained on Visual Genome.

The results, shown in the table above, include three categories of methods:

  1. Baseline methods using the same visual features and number of bounding region proposals.
  2. The model presented in this study.
  3. Non-comparable methods, including those using additional question-answer pairs from Visual Genome, multiple detectors, and model ensembles.

In Table 1, VisualBERT scores 70.80 on test-dev; Pythia v0.1 and v0.3, using comparable visual features and region counts, score 68.49 and 68.71. VisualBERT's test-std score is 71.00. The lower part of the table includes extra Visual Genome QA, multiple detectors or ensembles; MCAN configurations, for example, score 72.55, 72.80 and 75.00 on test-dev. The comparable-setting claim applies to the top baselines, not every row.

Ablation Study

This image is the paper's Table 5, not its VQA table or the main NLVR2 Table 3. For this ablation, every model uses only 36 region features per image, and the full model scores 66.7 on dev. Table 3's main result uses 144 proposals and scores 67.4 on dev. These two full-model scores have different input settings.

The study analyzed the contributions of the following four key components in VisualBERT:

  1. C1: Task-Agnostic Pre-training:

    • To examine the impact of task-agnostic pre-training, two variants were created:
      • Skip task-agnostic COCO pretraining (VisualBERT w/o COCO Pre-training), while retaining BERT initialization and subsequent training.
      • A model pre-trained with only text from COCO (without images) (VisualBERT w/o Grounded Pre-training).
    • Text-only COCO pre-training scores 63.9 on dev, and omitting COCO pre-training scores 62.9, both below the same-table full model's 66.7. This supports the value of paired vision-language pre-training in this setting.
  2. C2: Early Fusion:

    • Removing early fusion scores 61.4 versus 66.7 for the full 36-region model, a 5.3-point loss and the largest single ablation gap in this table.
  3. C3: BERT Initialization:

    • The full model starts with BERT weights. The variant without that initialization still receives subsequent COCO pre-training; it scores 64.7, 2.0 points below the full model. It is not a model without all pre-training.
  4. C4: Sentence-Image Prediction Objective:

    • Without sentence–image matching Objective 2, the dev score is 64.9, 1.8 points below the full model. It helps in this configuration, but the gap is smaller than those for C1 and C2.

In this 36-region NLVR2 ablation, paired task-agnostic pre-training (C1) and early fusion (C2) have the larger effects. This result does not rank their importance for every vision-language task or architecture.

Conclusion

VisualBERT shows strong task results, and its attention maps give us an intuitive glimpse of how words relate to image regions. These are useful observations of model behavior, though attention weights alone do not fully explain the model's decisions.

However, one challenge remains unavoidable:

  • The fusion Transformer is simple, but the full pipeline still needs object detection and region-feature extraction, adding deployment components and compute cost.
  • Overly intricate designs limit the model's potential for practical application and complicate deployment.

Therefore, optimizing and simplifying this architecture should be considered a key direction for future research.