Back to paper library
arXiv 2021ICML 2021Multimodal learningPaper notes

ViLT

Enter the Scene

Original paper

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Paper authors

Wonjae Kim · Bokyung Son · Ildoo Kim

Publication
ICML 2021Conference
First submitted to arXiv
arXiv:2102.03334
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityvilt2021,
  title = {{ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}},
  author = {Wonjae Kim and Bokyung Son and Ildoo Kim},
  year = {2021},
  eprint = {2102.03334},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.03334}
}
On this page

You might recall that the Transformer was introduced in 2017. Vision Transformer (ViT), specifically designed for images, appeared at the end of 2020, more than three years later.

This time, we don't have to wait that long. In less than six months, ViT was adapted to replace the visual architecture in VLP (Vision-Language Pretraining) models.

In recent years, research has focused on enhancing the visual representation capabilities of VL models to improve overall performance. But everyone seems reluctant to face one truth: pretrained object detection models are really heavy!

The authors of ViLT argue that we should abandon this architectural approach altogether!

Defining the Problem

The main issues and challenges defined by the authors can be summarized as follows:

  1. Complex Visual Encoding Process

    Current VLP models often employ deep convolutional networks or pretrained object detectors for visual encoding. This process is usually complex and computationally intensive, posing significant challenges in terms of computational resources and runtime, especially when dealing with large volumes of images.

  2. Heavy Visual Encoders

    Existing VLP models typically rely on heavy visual encoders to extract image features, placing a substantial burden on computational resources. Caching region features can mitigate this burden in academic experiments, but a new image in a live query still needs to be processed.

  3. Balancing Encoding Efficiency and Performance

    Striking a balance between maintaining model performance and improving encoding efficiency while reducing computational and storage costs has been a longstanding challenge. Most previous research has not focused on exploring lightweight and fast encoding methods that also deliver good performance on vision and language tasks.

  4. Limitations of Traditional Model Architectures

    Traditional VLP models often use deep, convolution-based encoding methods for visual inputs. This not only increases the model's complexity and runtime but also limits its flexibility and adaptability.

Solution

ViLT Design Philosophy

The authors categorize vision and language models along two dimensions:

  • Whether both modalities (vision and language) have equal levels of representation (dedicated parameters and/or computation).
  • Whether these modalities interact at deep network levels.

Combining these points, the authors identify four prototypes:

  • (a) Type: Vision Semantic Embedding (VSE) models, such as VSE++ and SCAN, have separate vision and language encoders, with heavier visual embedding; their interaction is relatively shallow, typically a dot product or shallow attention.
  • (b) Type: Models like CLIP, which use separate but computationally equal encoders for both modalities and interact at relatively shallow levels.
  • (c) Type: Deep interaction models, such as recent VLP models, utilize deep transformer models to model interactions between the two modalities at deep levels but still rely on computationally intensive convolutional neural networks for visual encoding.
  • (d) Type: ViLT model, which concentrates most computations on modality interaction modeling, keeping the encoding layer for raw pixels relatively shallow and lightweight, similar to handling text computations.

These four prototypes reflect different design and computational trade-offs:

  • (a) Type: Potentially computationally lighter but sacrifices deep modality interaction capabilities.
  • (b) Type: Distributes computational resources equally to vision and language encoders but maintains relatively shallow interactions.
  • (c) Type: Invests heavily in modality interactions but may overly rely on computationally intensive convolutional neural networks for visual input processing.
  • (d) Type: Concentrates most computation on deep interaction, while keeping the modality-specific visual and text embedding layers relatively light.

Each design choice brings different advantages and challenges. For instance, Type (b) models like CLIP excel in some zero-shot settings, but may be limited on tasks needing deep modality interaction. The paper's tested shallow-fusion NLVR2 setup is one example; the diagram does not say CLIP loses on every task. These categories give us a way to think about which design fits a specific application.

ViLT Design Details

ViLT attempts to simplify the visual encoding process using a patch projection strategy and initializes from ViT, which has visual feature processing capabilities. The overall model architecture emphasizes reducing computational complexity and resource requirements while maintaining performance, showcasing a lightweight multimodal learning strategy.

  1. Visual Encoding Method

ViLT shares a similar text encoding approach with other VLP models, typically leveraging BERT's tokenizers and encoding techniques but differs in visual encoding.

ViLT's design emphasizes simplicity in visual encoding, comparing several strategies:

  • Region Feature

    Traditional methods often use region features from object detectors (e.g., Faster R-CNN). ViLT avoids this approach, favoring more direct and simple methods.

  • Grid Feature

    Grid features are another method of visual feature extraction, typically using output feature grids from convolutional neural networks. However, ViLT does not primarily use this method.

  • Patch Projection

    ViLT adopts the patch projection method introduced by ViT. It cuts the image into 32×32 patches and linearly projects each patch to 768 dimensions, greatly simplifying visual encoding. This projection layer has about 2.4M parameters; the complete ViLT-B/32 model in Table 6 has 87.4M.

  1. Model Architecture

    • ViLT has a streamlined single-stream architecture: image patches and text tokens interact in a shared Transformer.
    • While many previous VLP models initialize from BERT weights, ViLT initializes the interaction Transformer from ViT-B/32. It uses the bert-base-uncased tokenizer, but learns the text embeddings from scratch. The authors cite earlier work reporting weaker results with BERT initialization; this paper does not present a separate BERT-initialization ablation of its own.
    • The ViT-B/32 weights were pretrained on ImageNet-21K, then fine-tuned on ImageNet-1K. ViLT subsequently performs image-text pretraining and downstream fine-tuning.
    • ViLT stacks include Multi-Head Self-Attention (MSA) and MLP layers, integrating textual information for multimodal pretraining.
  2. Encoding Details

    • Text input is processed with token embedding and positional embedding matrices.
    • Image input is divided into patches, flattened, linearly projected, and positionally encoded.
    • Text and image encodings each receive a modality type embedding, then are concatenated into one sequence and updated through multiple Transformer layers. The first pooled output feeds image-text matching; masked text outputs feed language modeling, while contextualized text and patch outputs feed alignment.

Pretraining Strategies

The pretraining objectives are image-text matching, word-patch alignment and masked language modeling. Whole word masking specifies how text is masked; image augmentation below is used in downstream fine-tuning.

  1. Image-Text Matching (ITM)

    • Random Image Replacement: To enhance the model's discriminative ability, there's a 0.5 probability of randomly replacing aligned images. In other words, there's a 50% chance that the image and its corresponding text description won't match.
    • ITM Head: A single linear layer projects the pooled feature into match/nonmatch logits, trained with negative log-likelihood loss.
  2. Word-Patch Alignment (WPA)

    Suppose an image shows a man eating an apple, with the text description "A man is eating an apple." Typically, the Image-Text Matching (ITM) objective focuses on the overall correlation between the entire image and the entire sentence.

    WPA instead encourages alignment at a finer level. It applies approximate optimal transport to the Transformer outputs for text tokens and image patches, then adds the approximate Wasserstein distance to the ITM loss with weight 0.1. The example suggests that apple should correspond to patches near the fruit, but there is no labeled "apple region" here.

  1. Masked Language Modeling (MLM)

    • Random Masking: Inspired by BERT, the authors randomly mask portions of the text and try to predict the masked parts based on context. There's a 0.15 probability of randomly masking text in the experiments.
    • MLM Head: The model uses a two-layer MLP to predict the masked tokens, aiming to minimize the negative log-likelihood loss between the prediction and the true tokens.
  2. Whole Word Masking

    • This strategy masks entire words rather than individual subwords. For example, "giraffe" might be tokenized as ["gi", "##raf", "##fe"]. Masking all its pieces prevents the model from guessing a missing piece from the others still visible; the authors expect this to encourage use of the surrounding context and image.
    • This approach is inspired by Pre-training with Whole Word Masking for Chinese BERT.
  3. Image Augmentation

    During downstream fine-tuning, the authors use RandAugment but exclude color inversion and cutout, not ordinary cropping. Color inversion might contradict words about color; cutout might erase a small but important object. The Table 5 augmentation result belongs to fine-tuning, not pretraining.

By combining these strategies, the authors aim to use the interaction between images and text more effectively without a heavy detector front end.

Discussion

In this study, the model is pretrained on several datasets, including MSCOCO, Visual Genome, SBU Captions, and Google Conceptual Captions, to capture rich image and language features.

During training, ViLT uses AdamW with learning rate warm-up and linear decay. Images are resized while preserving aspect ratio and sampled into patches; text uses the BERT tokenizer, with text-related parameters learned from scratch.

After pretraining, the model is evaluated on VQAv2 and NLVR2 classification and MSCOCO and Flickr30K retrieval. Classification scores are means of three runs with different seeds and data order, with standard deviations in the ablation table; retrieval is fine-tuned once.

This entire process is run on 64 NVIDIA V100 GPUs, covering 100K or 200K steps for pretraining and 10 epochs for training on downstream tasks, aiming to optimize performance on vision and language tasks.

Performance Analysis

For VQAv2, ViLT-B/32 scores 70.33 on test-dev, or 71.26 with fine-tuning augmentation and longer pretraining. The table lists 71.35 for Pixel-BERT-R50 and 75.95 for VinVL-Base. The authors suggest that separated object representations from a detector might make object-heavy VQA questions easier to learn. This remains an explanation, not a cause directly tested by this table.

For NLVR2, the model must check a sentence against two images. It pairs each image with the sentence, then concatenates the two pooled representations for binary prediction. ViLT-B/32 scores 74.41/74.57 on dev/test-P, or 75.70/76.13 after longer pretraining and fine-tuning augmentation. That is competitive with some entries, though below VinVL-Base's 82.05/83.08. Its measured inference time makes the trade-off interesting.

Key Contributions

The authors highlight three main operations based on the above table:

  1. More Training Steps

    In the first three rows, with no whole word masking, MPP or augmentation, increasing pretraining from 25K to 100K steps improves the listed metrics. In the later 100K-to-200K comparison, classification scores rise again, but fine-tuned Flickr30K and MSCOCO text retrieval R@1 fall from 83.69/62.88 to 83.50/61.49. So I cannot call the effect consistent across all metrics; the parenthesized zero-shot scores improve separately.

  2. Whole Word Masking

    At 100K steps, whole word masking raises VQAv2 from 70.16 to 70.33 and NLVR2 dev from 73.54 to 74.41. That supports its effect on these tasks, not a claim about language generation, which was not evaluated. Table 5 also shows that adding masked patch prediction (MPP) worsens most reported metrics; NLVR2 test-P, for example, falls from 74.57 to 73.54.

  3. Image Augmentation

    Adding RandAugment during fine-tuning raises the 100K-step model's VQAv2 from 70.33 to 70.85 and NLVR2 dev from 74.41 to 74.91. The parenthesized zero-shot retrieval values do not change in this comparison.

Complexity Analysis

Complexity analysis of VLP models can be approached from several angles, including parameter count, FLOP count, and inference latency. Key points of detailed discussion include:

  1. Parameter Count

    Excluding text embeddings, Table 6 lists 87.4M parameters for ViLT-B/32, 94.9M for Pixel-BERT-R50, and 154.7M for UNITER-Base. Parameter count describes size.

  2. FLOP Count

    FLOPs estimate forward-pass work. Table 6 lists 55.9G for ViLT-B/32, 136.8G for Pixel-BERT-R50, and 949.9G for UNITER-Base. Image resolutions and token counts differ, so this is not a same-input architecture ablation.

  3. Inference Latency

    The authors measured visual embedding plus Transformer inference on a Xeon E5-2650 CPU and NVIDIA P40 GPU: about 15 ms for ViLT-B/32, 60 ms for Pixel-BERT-R50, and 900 ms for UNITER-Base. Text embedding is excluded. Per-class NMS can add latency to region methods even though it is not represented by tensor FLOPs. These benchmark numbers do not promise the same service latency on other hardware.

  4. Input Resolution and Length

    Different VLP models use different image resolutions, likely based on the design objectives and application scenarios. Higher resolutions typically mean more detail but also higher computational costs.

  5. Processing Input Sequences

    On this hardware, a BERT-base-style Transformer took less than 1 ms longer for input sequences under length 300. Table 6 uses 200 image patches plus 40 text tokens for ViLT-B/32; image patches can reach 240, and the total then still includes text tokens.

Considering these aspects, the complexity of VLP models is multifaceted, involving computational cost, resource usage, application scenarios, and performance. Understanding these sources and impacts of complexity provides crucial guidance for model selection, optimization, and deployment in practical applications.

Conclusion

At the time of this paper's publication, ViT had just been introduced, and many theories and practices were still being explored. Following ViT's introduction, numerous discussions and improvement papers emerged, focusing on training efficiency, accuracy improvements, and related research and theoretical debates for an extended period.

However, ViLT also showed competitive performance in some tasks and striking measured inference speed. For real-time applications, that speed could be valuable, though the paper's hardware benchmark is not an end-to-end deployment test.

ViLT's proposal not only provides new directions for current model design and pretraining but also clearly indicates that future work should focus more on internal modality interactions within models rather than solely expanding model size or increasing network complexity. This presents a novel approach, pointing toward the potential new direction for model development: achieving performance improvements through more sophisticated design without adding significant computational burden.

Such models point to a future focused on modality interaction and internal structure optimization. Future research can delve deeper into ViLT's potential and limitations, drawing inspiration to develop more innovative models and application strategies.