Back to paper library
arXiv 2021CVPR 2022Multimodal learningPaper notes

METER

Vibrant Dashboard

Original paper

An Empirical Study of Training End-to-End Vision-and-Language Transformers

Paper authors

Zi-Yi Dou · Yichong Xu · Zhe Gan …

Show all 12 authors

Zi-Yi Dou · Yichong Xu · Zhe Gan · Jianfeng Wang · Shuohang Wang · Lijuan Wang · Chenguang Zhu · Pengchuan Zhang · Lu Yuan · Nanyun Peng · Zicheng Liu · Michael Zeng

Publication
CVPR 2022Conference
First submitted to arXiv
arXiv:2111.02387
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalitymeter2021,
  title = {{An Empirical Study of Training End-to-End Vision-and-Language Transformers}},
  author = {Zi-Yi Dou and Yichong Xu and Zhe Gan and Jianfeng Wang and Shuohang Wang and Lijuan Wang and Chenguang Zhu and Pengchuan Zhang and Lu Yuan and Nanyun Peng and Zicheng Liu and Michael Zeng},
  year = {2021},
  eprint = {2111.02387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.02387}
}
On this page

This is an empirical paper that brings many VLP design choices together for comparison.

So, you can expect to see experiments, and even more experiments, here.

Defining the Problem

The current mainstream Vision-Language Pretraining (VLP) architectures generally consist of three components:

  • First, the vision encoder, commonly referred to as ViT.
  • Second, the text encoder, most frequently BERT.
  • Finally, the co-encoder, which integrates visual and textual information.

The authors first compiled a summary of architectures used in previous research, then deconstructed and recombined them.

Ultimately, it all boils down to issues with the visual component:

First, inefficiency.

In many VLP studies, most models rely on object detectors (e.g., Faster RCNN) to extract region features. These detectors are often frozen during the VLP process, which limits the capacity of VLP models. Additionally, the feature extraction process is time-consuming, which can affect the efficiency and practicality of the models.

Second, insufficient exploration. Fully transformer-based VLP models have yet to be thoroughly explored. Some ViT-based models still lag behind state-of-the-art performance in downstream tasks like visual question answering.

Finally, optimizer inconsistency. Some CNN grid-feature systems use different optimizers for their CNN and Transformer components, making end-to-end training less straightforward. The paper uses this as one motivation to examine fully Transformer-based VLP.

Solving the Problem

METER Model Design

The authors broke the test units into several components: vision encoder, text encoder, multimodal fusion module, architecture design, and pretraining objectives.

  1. Vision Encoder Selection

    In the ViT mechanism, images are first divided into multiple patches, which are then processed by a Transformer model.

    The authors compared various ViT models, including the following:

  2. Text Encoder Selection

    The authors aimed to process text features with a text encoder before sending them to the fusion module. They explored using different language models like BERT, RoBERTa, ELECTRA, ALBERT, and DeBERTa for text encoding. Additionally, they experimented with a simple word embedding lookup layer initialized with BERT’s embedding layer.

    Below are the architectures selected by the authors:

  3. Multimodal Architecture

    As VLP models gain popularity, the authors re-evaluated the effects of two fusion modules in a new environment to determine which fusion strategy is more effective:

    • Merged Attention Module: In this module, text and visual features are simply concatenated and jointly input into a single Transformer module. This approach enables simultaneous processing of text and visual information in the same Transformer module.
    • Co-attention Module: This module independently inputs text and visual features into separate Transformer modules. It uses cross-attention mechanisms to achieve interaction between modalities, enabling interaction between visual and textual features.

Encoder-Only vs. Encoder-Decoder

The authors compared two different model architectures: encoder-only and encoder-decoder, analyzing their applications in VLP models:

  • Encoder-Only Architecture: In this architecture, cross-modal representations (e.g., combined visual and text features) are directly input to the output layer to generate the final output. VisualBERT is an example of a VLP model employing an encoder-only architecture.

  • Encoder-Decoder Architecture: Prominent works like VL-T5 and SimVLM advocate for an encoder-decoder structure. Here, cross-modal representations are first passed through a decoder, which then generates outputs by attending to both encoder representations and previously generated tokens in an autoregressive manner.

The diagram illustrates their masked-language-modeling workflows. For the Visual Question Answering (VQA) classification task:

  • In the encoder-decoder model, the text input is passed to the encoder, and classification tokens are passed to the decoder.
  • In the encoder-only model, cross-modal representations are directly fed into the output layer.

Pretraining Strategies

The authors identified three primary pretraining tasks commonly used in VLP models: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Image Modeling (MIM):

  1. Masked Language Modeling (MLM) Initially introduced in pure language pretraining, MLM was later adapted for VLP. The goal is to randomly mask some input tokens in image-caption pairs and train the model to reconstruct the masked tokens.

  2. Image-Text Matching (ITM) This task requires the model to identify which images and captions match, typically framed as a binary classification problem. The model learns global cross-modal representations and uses a classifier to predict whether the pairs match.

  3. Masked Image Modeling (MIM) This task predicts hidden image content. The paper tests two masked-patch classification variants: one uses other patches in the batch as negatives; the other predicts discrete visual codes. These are ablations; the default METER pretraining uses only MLM and ITM.

Dataset

The model was pretrained on four commonly used datasets: COCO, Conceptual Captions, SBU Captions, and Visual Genome. Following previous studies, the combined training data consists of approximately 4 million images.

Technical Details

  • Hidden Size: Unless otherwise specified, the hidden layer size is set to 768, and the number of attention heads is 12.
  • Separate Visual and Language Branches: There is no decoder between the visual and language branches, and no parameter sharing between them.
  • Pretraining Tasks: The model is pretrained using only MLM and ITM, except in special cases.
  • Optimizer: The model was pretrained for 100,000 steps using the AdamW optimizer. The learning rates for the lower and upper layers were set to 1e-5 and 5e-5, respectively.
  • Warmup Ratio: The learning rate warms up over the first 10% of training steps, then linearly decays to 0 over the remaining steps.
  • Image Resolution: The size of each image is adjusted to 224×224 or 384×384.

Discussion

Evaluating Components Separately

Due to the time-intensive nature of pretraining, the study first evaluates different text and vision encoders without performing full VLP to improve efficiency. These models are still fine-tuned on the downstream tasks:

  • The lower layers of the model are initialized with pretrained vision and text encoders, while the upper layers are randomly initialized.
  • The default encoders chosen are CLIP-ViT-224/32 for vision and RoBERTa for text, where ViT-N/M denotes the image resolution (N) and patch size (M).
  1. Text Encoder Evaluation

    The performance gaps among the pretrained text encoders are modest. RoBERTa is steady across several metrics, but ALBERT has the highest VQA score, and other encoders lead some retrieval columns. The lower VQA and VE scores of Emb-only suggest a pretrained text encoder helps in this setup.

  2. Vision Encoder Evaluation

    Both CLIP-ViT-224/16 and Swin Transformer perform well in this setup. Notably, the Swin Transformer reaches 72.38 on VQA test-dev without image-text pretraining but after VQA fine-tuning. This is not a zero-shot VQA result, though it is competitive with some VLP models.

V+L Combined Evaluation

When combined into a VLP architecture, the differences between BERT and RoBERTa narrow, though having a pretrained text encoder remains critical (as shown in the comparison between Embed-only and RoBERTa).

For the vision encoder, both CLIP-ViT-224/16 and Swin Transformer deliver strong performance. Specifically, CLIP-ViT-224/16 reaches 77.19/77.20 on VQA test-dev/test-std, outperforming the paper's listed VinVL comparison. The pictured Table 4 shows 77.19 test-dev; 77.20 test-std comes from the paper's accompanying text.

Feature Fusion Analysis

The authors designed Merged Attention and Co-attention models and compared their performance.

With roughly matched parameter counts, the Co-attention model scores higher than Merged Attention on VQA and Flickr zero-shot retrieval in this experiment. Separate modality parameters seem helpful in this setup; the comparison alone does not settle every VLP architecture. This finding differs from earlier region-based VLP models. The authors offer two possible reasons:

  1. Results from region-based VLP models may not directly apply to ViT-based VLP models.
  2. Most region-based VLP models use pretrained vision encoders without pretrained text encoders, making symmetric architectures like Co-attention less applicable in those scenarios.

On the other hand, in the experiment comparing Encoder and Encoder-Decoder architectures, the authors followed T5-style language modeling objectives:

  • Masked 15% of input text tokens.
  • Replaced continuous spans of text with sentinel tokens.
  • Trained the decoder to reconstruct the masked tokens.

For Image-Text Matching (ITM) tasks, special classification tokens were fed into the decoder to generate binary outputs.

The table favors the Encoder-only model on VQA test-dev (74.98 versus 74.73) and Flickr zero-shot retrieval (IR@1 66.08 versus 48.96, TR@1 78.10 versus 71.60). A decoder is convenient for generative tasks such as image captioning.

Expanding Dataset Scale

The authors tested the scalability of their framework by pretraining the model with more images and a larger vision backbone. The specific pretraining datasets include COCO, CC, CC12M, SBU, and VG, providing a total of approximately 14 million images and 20 million image-caption pairs, offering rich data for pretraining.

For the vision backbone, they adopted CoSwin-Huge, which is capable of handling large-scale visual data. On the text side, they used RoBERTa-base to process textual information, ensuring effective text encoding.

Under this expanded configuration, METER-CoSwin-Huge reaches 80.33 on VQAv2 test-dev and 80.54 on test-std, slightly above the table's SimVLM-Huge row (80.03 / 80.34). SimVLM's cited scale is 1.8 billion image-text pairs. Because this METER experiment changes both data scale and visual backbone, it cannot isolate the benefit of either one.

Conclusion

This paper systematically studies how to train a full-transformer VLP model end-to-end. The experiments demonstrate that competitive performance can be achieved with just 4 million images for pretraining using state-of-the-art models. Moreover, when scaled up, METER achieved new state-of-the-art performance on VQA, demonstrating its strong potential for further improvement.