Unifying Vision-and-Language Tasks via Text Generation
- Publication
- ICML 2021Conference
- First submitted to arXiv
- arXiv:2102.02779
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityvlt52021,
title = {{Unifying Vision-and-Language Tasks via Text Generation}},
author = {Jaemin Cho and Jie Lei and Hao Tan and Mohit Bansal},
year = {2021},
eprint = {2102.02779},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2102.02779}
}
After reading several papers, you might be familiar with some of the prediction methods for various downstream tasks. Let's look at two examples:
The first example is VQA, generally executed as shown in the above image (a):
-
Object Detection: First, obtain object results from an object detector in the image. This step helps identify and locate the main objects and features in the image.
-
Token Input: Convert the results from the object detector into tokens and place them at the beginning of the input sequence. Then, convert the question into tokens and place them after the image tokens in the input sequence.
-
Transformer Model: Feed the combined token sequence of the image and question into the Transformer model. The Transformer model will perform deep self-attention operations, capturing contextual information and generating an encoded representation of the entire input sequence.
-
Prediction Head and Sigmoid Function: Pass the output of the Transformer model (usually the encoded representation of the [CLS] token) to a specialized prediction head. This head will calculate a score for each possible answer. Each score is then passed through a sigmoid function to compress it between 0 and 1, which can be interpreted as the model's confidence that the answer is correct.
-
Select an Answer: The model can rank the highest-scoring answers internally, then normally selects one final answer. It is constrained to the predefined answer vocabulary.
Many VQA datasets, especially those using multiple-choice answers, may be limited by predefined answer sets. This means that even if a more suitable answer exists, the model might miss certain details or information if the answer is not in the predefined set. Additionally, there can be multiple reasonable answers to some questions, which may impact the model's performance during evaluation, even if the provided answer is reasonable in some contexts.
The second example, Figure 3(b), is referring-expression comprehension or visual grounding: given a phrase such as “fire hydrant,” choose its region in the same image. The steps are similar at first:
-
Object Detection: First, obtain object results from an object detector in the image to identify and locate the main objects and features.
-
Token Input: Convert region features to visual tokens and combine them with tokens for the referring expression.
-
Transformer Model: Feed the combined tokens to the Transformer to obtain contextual representations of the expression and candidate regions.
-
Softmax Prediction Head: Score the candidate region representations and normalize those scores.
-
Select a Region: Choose the matching region from this image, not one of several images.
This approach requires an additional region-scoring head, which the authors find cumbersome. VQA and grounding have different heads despite sharing much of the underlying vision-and-language work.
Defining the Problem
The authors argue that current methods require task-specific architectures and objectives for handling various vision-and-language tasks, which increases model complexity and limits generalization and efficiency:
-
Unified Task Architecture
- Traditional vision-and-language Transformer models often require task-specific, separately parameterized architectures for different pretraining or downstream tasks. This makes model design and fine-tuning relatively complex.
- Whenever a new task arises, the model must be redesigned or fine-tuned, increasing workload and complexity.
- Many skills required for vision-and-language tasks overlap significantly.
-
Open-Ended Answer Generation
- Different tasks require different ways to represent their labels. For example, existing visual question-answering methods treat answers as multi-label classification problems within a fixed set, although answers are already text.
- Discriminative methods are limited when answering open-ended questions because they can only choose from a predefined set of answers, rather than generating open-ended natural language answers.
-
Multi-Task Learning and Generalization
- Current pretraining methods need task-specific architectures and objectives, which may limit their generalization and efficiency.
- Traditional methods struggle with rare answers during training, particularly in visual question answering, affecting their generalization ability.
Solution
VL-T5 Model Design
VL-T5 aims to unify vision and language tasks into multimodal conditional text generation. The key design points are:
-
Based on Pretrained Transformer Models
- VL-T5 starts from T5-Base. The same paper also builds VL-BART from BART-Base.
- To handle visual input, the authors extended the text encoder by incorporating image region encodings as additional inputs to create a multimodal encoder.
-
Visual Embeddings
- Derived from Faster R-CNN image representations using 36 object regions to represent the input image.
- Each image region's features are a combination of four types of information: RoI object features, RoI bounding box coordinates, image ID, and region ID.
- Linear layers encode the RoI feature and box coordinates; learned embeddings encode image IDs and region IDs. Image IDs distinguish the two images in tasks such as NLVR2.
-
Text Embeddings
- To accommodate various tasks without designing task-specific architectures, the model adds specific prefixes to the original input text.
- Text embeddings are shared across encoder, decoder and language-model head. Visual markers such as
<vis_3>correspond to region IDs and reuse their text-token embeddings, connecting input regions with generated labels.
-
Encoder-Decoder Architecture
- Uses a Transformer encoder-decoder architecture to encode visual and text inputs and generate text labels.
- The encoder accepts concatenated text and visual embeddings as input, then outputs their joint contextual representations.
- The decoder focuses on previously generated tokens and encoder outputs to generate text.
-
Consistent Output Format
-
Unified Framework vs. Task-Specific Methods
- Traditional methods often develop specialized architectures and objectives for specific tasks.
- VL-T5 provides a unified framework that does not require redesigning models for each new task.
-
Visual Question Answering
- Traditional models typically introduce a multilayer perceptron (MLP) multi-label classifier head for visual question answering. These models are trained with binary cross-entropy loss alongside the Transformer backbone, weighted by VQA scores.
- VL-T5 addresses visual question answering by treating both questions and answers as text and using the same language modeling objective.
-
Grounded Expression Understanding
- Traditional methods (e.g., UNITER) solve this problem by adding an MLP region scoring head to the output representations of image regions, typically involving multi-class or binary classification strategies.
- VL-T5 instead generates the target region's ID as a visual token such as
<vis_3>using the same language-modeling objective. It still relies on detector-proposed candidate regions.
-
Datasets Used
-
MS COCO
This popular dataset is primarily used for object detection, semantic pixel segmentation, and image captioning tasks. "COCO" stands for "Common Objects in Context," indicating that the images include objects within real-world contexts. It is widely used in deep learning and computer vision for training and validation.
-
Visual Genome (VG)
VG is another popular dataset designed to provide detailed visual knowledge by deeply understanding objects, attributes, and relationships within images. It not only annotates objects but also includes interactions and relationships between them.
-
Dataset Scale and Details
- The pretraining data include about 180,000 distinct images from MS COCO and Visual Genome. Alongside captions, the VQA pretraining task uses questions from VQA v2.0, GQA and Visual7W.
- The authors count 9.18 million image-text pairs across this multi-task pretraining set, not 9.18 million different images.
Pretraining Tasks
-
Multimodal Language Modeling (MLM)
VL-T5 is a multimodal version based on the T5 (Text-to-Text Transfer Transformer) architecture.
VL-T5 selects 15% of input text tokens for masking, replaces each contiguous masked span with its own sentinel token, and generates the missing spans. VL-BART instead masks 30% of text with its
<mask>scheme. This is one of several pretraining tasks, not the entire training objective.I am not completely sure how best to translate “sentinel token”; literally, it is a marker standing in for a missing span. It appears in the target sequence again, followed by the text removed from that span. This paper's Table 1 uses tokens like
<text_1>, while generic T5 examples often use[extra_id_1].
The point is to mark and restore missing spans in a text-to-text setup. Unlike BERT's repeated [MASK], each span receives its own numbered marker, so the decoder can generate the missing text in order.
BERT's [MASK] trains a model to predict masked tokens from context. The sentinel design fits T5's text-to-text format, where the model generates a target sequence containing the missing spans.
Although sentinel tokens are fixed, their specific content is dynamic, depending on the current context and the model's predictions. In different contexts, the same sentinel token can be used without ambiguity, as its meaning is explicitly interpreted and restored during the model's training or prediction process.
-
Visual Question Answering (VQA)
Traditional VQA methods often have a predefined set of answer options (e.g., a fixed answer vocabulary), and the model's task is to choose the most suitable answer from them. This approach is limited because it cannot generate answers not present in the predefined set. However, the authors' proposed method allows the model to directly generate the original text of the answer, offering greater flexibility to answer more open-ended questions rather than just selecting predefined answers.
-
Image-Text Matching
This task involves determining whether an image and a piece of text are related or matched. For example, an image might show a dog playing in a park, and the corresponding text might be "A dog chasing a ball on the grass." If these elements (image and text) describe the same content or context to some extent, they are considered matched.
Here the authors use COCO captions and two kinds of pairs:
- Correct Pair: This is the original, true image-text pair, indicating they are matched in the real world.
- Incorrect Pair: With probability 0.5, a caption is drawn from a different training image and paired with the current image. The changed item is the caption, not the image.
The model's task is to predict the relationship between each image and text pair, determining whether they are "true" (they match) or "false" (they don't match). This is essentially a binary classification problem, where the model learns to identify features that indicate true matching pairs from the image and text.
-
Visual Grounding
Visual grounding, or visual localization, asks the model to find a visual region corresponding to a text description. In this pretraining task, the authors select a detector-predicted region and form a description from its predicted object name and attributes. Given that text, the model generates the region's visual ID token, such as
<vis_3>.This pretraining task needs no extra human word-to-region labels; it still depends on the detector's predicted regions and descriptions.
-
Grounded Captioning
Grounded captioning generates a description for one region rather than for the entire image. Think of it as the reverse of visual grounding above: given a region's visual token, generate its object-and-attribute description.
-
Pretraining Execution Details
- Pretraining was conducted for 30 epochs using 4 RTX 2080 Ti GPUs, taking about 4 days. The reported batch sizes are 320 for VL-T5 and 600 for VL-BART.
- AdamW was used as the optimizer with specific hyperparameter settings.
Discussion
VQA and GQA Analysis
VL-T5 and VL-BART were compared with several existing vision-and-language pretrained Transformers across seven downstream tasks. Unified generation approaches some task-specific models on some tasks, but the columns differ. For VL-T5, the table reports 70.3 VQA, 60.8 GQA, 73.6 NLVR2, 71.3 RefCOCOg, 58.9 VCR, 116.5 COCO captioning, and 38.6 Multi30K translation. The first five are accuracies, followed by CIDEr and BLEU; they are not interchangeable metrics.
VL-BART reaches 71.3 on VQA test-std, slightly above VL-T5's 70.3, but trails on RefCOCOg and VCR. Its RefCOCOg test score is only 22.4 against VL-T5's 71.3. The discussion below concentrates on VL-T5.
-
Performance in VQA and GQA
Visual question answering asks for an answer based on the given image. VL-T5 scores 70.3/60.8 on VQA/GQA test-std; VL-BART scores 71.3/60.5. LXMERT, with the same 180K pretraining-image count, scores 72.5/60.3. These results are close, but neither generative model wins both comparisons.
-
Generative Models vs. Discriminative Models
Most methods treat VQA as multi-label classification over a fixed answer list. The authors divide questions by whether the best answer falls within 3,129 frequent candidates. Outside that set, generative VL-T5 scores 13.1, compared with 7.1 for a discriminative head on the same backbone and 10.0 for UNITER-Base. Yet UNITER-Base scores 70.5 overall, above VL-T5's 67.9. The advantage shown is for this rare-answer group, not every unseen question or the whole benchmark.
-
Effect of Dataset-Specific Prefixes
Different text prefixes can change fine-tuning results. Using one
vqaprefix for both VQA and GQA, rather than a separate prefix per dataset, raises VQA Karpathy-test from 67.9 to 69.3 and GQA test-dev from 60.0 to 60.2 in the tested setup.
NLVR2 Analysis
The NLVR2 task requires the model to determine whether a natural language statement applies to a pair of images. This task tests the model's ability to integrate image and text information. It is divided into three encoding settings:
- Triplet: This setting jointly encodes the image pair and corresponding text.
- Pair: This setting individually encodes each image with its corresponding text before concatenation.
- Pair-biattn: Based on the Pair setting, it adds bidirectional attention.
VL-T5 uses Triplet: it concatenates the two images' region features, distinguishes their origins with image-ID embeddings, and generates true or false. Its dev/test-P scores are 74.6/73.6. UNITER with the same Triplet setup scores 73.0/73.9; UNITER Pair-biattn scores 77.2/77.9. Pair and Pair-biattn take about twice Triplet's computation, so the cross-setup gap cannot be attributed solely to the model architecture. The simple Triplet result is still useful evidence for the unified format.
VCR Analysis
VCR (Visual Commonsense Reasoning) is a multiple-choice answering task requiring commonsense reasoning beyond simple object or action recognition. Each VCR question has four answer options and four rationale options. This task can be broken down into two sub-tasks: Question Answering (Q→A) and Answer Justification (QA→R). The overall goal is to choose the correct answer and provide the correct rationale for that answer.
VL-T5's approach is similar to that of Nogueira et al. The model concatenates the context (image and question) with each candidate option and generates true or false. During inference, it ranks candidates by and chooses the highest score.
On the VCR validation set, VL-T5's Q→AR score rises from 52.5 without pretraining to 54.7 after generic pretraining and 57.5 after a second, VCR-domain pretraining stage. The two-stage test model scores 58.9, close to the comparably two-stage UNITER's 58.2 and above ViLBERT's 54.8. Keep those training stages in view when comparing the models.
Conclusion
The Encoder-Decoder architecture of Transformers comes up often. VL-T5 explores it deeply for vision-and-language tasks, with the practical aim of using one output format. Yet these first results are not a leap ahead on every benchmark.
While the combination of Encoder and Decoder has proven effective in many tasks, in specific vision-and-language integration tasks, it may still require further adjustment and optimization. This might involve more precise architectural adjustments, different attention mechanisms, or more suitable pretraining strategies for specific tasks.
Therefore, simply changing or adjusting the model architecture may not be sufficient to bring significant performance improvements. Perhaps more importantly, how to fully utilize large amounts of data, more effective training strategies, and deeper task understanding to optimize for specific problems.
From our own development experience, an Encoder-only setup is often simpler to train for a task that needs an input representation. It can be enough to encode the useful features and attach a prediction head. A full Encoder-Decoder also has to learn how to generate an output sequence, adding training and decoding work; the total speed and parameter cost still depend on the actual design.
That makes me wonder whether generation here might benefit from much more data. A Decoder learns an output sequence conditioned on the Encoder's representation, which is a different training problem from selecting among fixed labels. More data could help with rare outputs; the paper evaluates this question with roughly 180K images.
In summary, VL-T5 provides a valuable starting point, offering new insights into the application of the Encoder-Decoder architecture in vision-and-language integration. By exploring the potential limitations and strengths of this architecture, and further adjusting and optimizing it, we can achieve higher performance while paving new paths for researchers in this field.