MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
- Publication
- ICCV 2021Conference
- First submitted to arXiv
- arXiv:2104.12763
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalitymdetr2021,
title = {{MDETR - Modulated Detection for End-to-End Multi-Modal Understanding}},
author = {Aishwarya Kamath and Mannat Singh and Yann LeCun and Gabriel Synnaeve and Ishan Misra and Nicolas Carion},
year = {2021},
eprint = {2104.12763},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2104.12763}
}
In recent years, object detection has been at the forefront of computer vision, serving as the core of many advanced multi-modal understanding systems. However, traditional methods treat detection systems as black boxes, conducting fixed concept image detection, which clearly has inherent limitations.
A significant issue is the inability to effectively utilize multi-modal context for collaborative training, limiting downstream models to accessing only detected objects. Moreover, these detection systems are often frozen, meaning they lack further refinement and adaptability. Most importantly, the vocabulary of these detection systems is severely limited, making them blind to novel concept combinations expressed in free-form text.
Simply put, the authors of this paper aim to replace the object detection architecture in VL models.
You might remember a previous paper that attempted to replace the detection architecture with ViT, which didn’t turn out well.
- Portal: ViLT: Passing the Baton
Since replacing with ViT wasn’t very successful (because ViT isn’t specialized for object detection), this time the authors used another "specialist" object detection model, DETR, for the replacement.
- Portal: DETR: The Pioneer Across Fields
Problem Definition
Most advanced multi-modal understanding systems currently rely on object detection as a core component, but these designs are fraught with various issues:
-
Limitations in Collaborative Training
In multi-modal systems, collaborative training means using data from multiple input sources (e.g., images, text, audio) simultaneously for model training. If a system component cannot collaborate with other parts for such training, it might fail to fully leverage all available information.
Imagine a model with both image and audio inputs that only trains the image detector independently, ignoring the audio input. If the audio provides important information about objects in the image, the model might fail to identify them correctly.
-
Detection Scope Limitation
The main purpose of detection systems is to identify specific objects in images. However, if these systems focus solely on known objects and ignore other parts of the image, they might miss important contextual information. For instance, in an image with multiple people and a dog, the detector might only identify the people and dog, ignoring the park setting, which provides important context.
-
Model Fixation
Once a model is trained and "frozen," it no longer updates or learns. This hinders its ability to adapt and optimize in new situations or with new data. For example, a detector trained on summer images might perform poorly on winter images with snow or people in heavy coats if it cannot be fine-tuned.
-
Vocabulary Limitation
Object detection systems identify specific classes or attributes based on their training data. If they encounter new objects or concepts not seen in the training data, they might fail to recognize them.
-
Lack of End-to-End Design
End-to-end systems allow learning and optimization from input to output without a fixed intermediate detector. If that detector is frozen or its predictions are detached from the downstream loss, the loss cannot update it through backpropagation. The detector may itself be differentiable, but it cannot learn from that downstream task in this setup.
Solution
MDETR Model Design
The model is straightforward: for the text part, it uses a modified RoBERTa encoder. Image and text features are first projected to the same dimension, then concatenated along the sequence dimension and passed through a joint DETR-style Transformer encoder and object-query decoder.
The overall structure includes:
- Main Convolutional Encoder: The image is first encoded by a convolutional backbone and flattened.
- Spatial Information: By adding 2D positional encoding to the flattened vectors, the model retains spatial information.
- Text Encoding: The text is encoded using a pretrained Transformer language model, producing a sequence of hidden vectors of the same size as the input.
- Modality-Specific Linear Projections: Modality-specific linear projections are applied to image and text features to project them into a shared encoding space.
- Cross-Encoder: The concatenated sequence of image and text features is fed into a joint Transformer encoder, the core part of the model.
- Object-Query Decoder: As in DETR, object queries attend to the jointly encoded sequence before predicting boxes and their associated text.
Training Method
- Soft Token Prediction
The idea behind soft token prediction is intriguing. For each object query matched to a ground-truth box, it predicts a distribution over corresponding text-token positions rather than a fixed object class. Tokens in the referring span receive uniform target probability; unmatched queries learn the "no object" target. This is a main difference from standard object detection.
Suppose the description sentence is "a black cat and a white dog." When the model detects a black animal, it tries to predict its association with the phrase "black cat." This is not just about an individual token or class label but about a range of tokens in the text that collectively describe a specific object.
The benefit of this approach is that it can handle multiple references to objects in the same text or cases where multiple objects correspond to the same text.
Contrastive Alignment
Contrastive alignment aims to ensure that the encoded representations of visual objects are close to their corresponding text tokens in the feature space. This alignment is stronger than just relying on positional information from "Soft Token Prediction" as it directly operates on feature representations.
Referencing the mathematical equation provided in the paper:
- L: The maximum number of tokens.
- N: The maximum number of objects.
- T+i: The set of tokens that should align with a given object oi.
- Oi+: The set of objects that should align with a given token ti.
- τ: The temperature parameter, set to 0.07 directly.
The pictured equation covers object-to-token alignment. An object may have several positive tokens, so the positive terms are averaged. The paper also defines the symmetric token-to-object direction; the mean of both is the full contrastive alignment loss. The idea is simple: objects and text should line up in feature space—the closer the match, the better.
All Losses
MDETR first uses bipartite matching to assign predicted boxes to ground truth, with matching costs for soft tokens, L1 box distance, and GIoU. After matching, the training loss combines box L1/GIoU, soft-token cross-entropy, and the contrastive alignment above. Bipartite matching is an assignment step, not an extra box-prediction loss.
Dataset
- CLEVR: Used to evaluate the method's results.
- Flickr30k: Used to build the composite dataset images.
- MS COCO: Used to build the composite dataset images.
- Visual Genome (VG): Used to build the composite dataset images.
- Annotation data is sourced from referring expression datasets, VG regions, Flickr entities, and the GQA training balanced set.
Technical Details
-
Pre-training Modulated Detection
During the pre-training phase, the goal is to detect all objects referenced in the aligned free-form text.
-
Data Composition Technique and Its Importance
- For each image, all relevant text annotations are retrieved from the mentioned datasets. If different annotations reference the same image, they are merged. To ensure the independence of the training and validation/test sets, all images that appear in the validation or test sets are removed from the training set.
- A graph-coloring algorithm combines sentences whose referred-to bounding boxes have GIoU ≤ 0.5; this threshold is not about text-block overlap. The combined sentence is limited to fewer than 250 characters. From about 200,000 images, this process forms 1.3 million aligned image–text pairs. Those counts refer to different units.
- There are two main reasons for using this data composition technique:
- Data efficiency: By packing more information into a single training sample, the data can be utilized more effectively.
- Better learning signal:
- When the model learns, it needs to identify and resolve ambiguities between multiple references to the same object class in the text.
- In a single-sentence context, the task of "Soft Token Prediction" becomes relatively simple, as the model can usually easily predict the subject or core meaning of the sentence without much reliance on the image.
- By combining multiple sentences, the model is forced to explore the associations between images and text more deeply, thereby improving its prediction capabilities.
-
Model Architecture Choices
- The text encoder uses a pretrained RoBERTa-base with 12 Transformer encoder layers.
- Visual backbones include:
- ResNet-101: Obtained from Torchvision and pretrained on ImageNet.
- EfficientNet Series: EfficientNetB3 and EfficientNetB5 were used. EfficientNetB3 achieved 84.1% top-1 accuracy on ImageNet, while EfficientNetB5 achieved 86.1%.
- The EfficientNet-B3/B5 backbone weights came from Noisy-Student pseudo-label pre-training with ImageNet and large amounts of unlabeled data; this is not another independently added MDETR branch.
- Training details: Using 32 V100 GPUs, pretraining was conducted for 40 epochs with an effective batch size of 64, taking approximately one week.
Discussion
Downstream Task Performance Analysis
-
Phrase Grounding
The authors used the Flickr30k dataset with specific training/validation/test splits. For evaluation, they adopted two different protocols to address the issue of multiple objects being referenced by a single phrase, each with its pros and cons.
-
ANY-BOX Protocol
If a phrase refers to several objects, Recall@ counts a hit when at least one of the top predicted boxes has IoU above 0.5 with any corresponding ground-truth box. It does not ask whether every referenced instance was found.
-
MERGED-BOXES Protocol
If a phrase refers to several objects, their ground-truth boxes are enclosed by one merged box, then IoU is calculated against it. Since MDETR normally predicts separate instance boxes, the paper fine-tunes separately on merged-box annotations for this protocol; it is not simply the same output scored two ways. Merging can also hide individual instances, especially when they are far apart.
-
Results Comparison
- Under ANY-BOX, MDETR-R101 without additional image–text-pair pre-training reaches 78.9 validation Recall@1, 8.5 percentage points above VisualBERT† at 70.4. This does not erase the existing pre-training of RoBERTa or the image backbone, and the methods do not have identical conditions.
- After COCO, VG, and Flickr30k pre-training, MDETR-R101 reaches 83.4 test Recall@1, 12.1 points above the ResNet-101-based VisualBERT† at 71.3. The ENB5 best row uses a different backbone again.
-
-
Referring Expression Comprehension
Most previous research and methods for this task involve ranking a set of pre-extracted image-related bounding boxes. These boxes are obtained using pretrained object detectors.
In contrast, this paper aims for a more challenging goal: training the model to predict bounding boxes directly given a referring expression and the corresponding image, rather than simply ranking pre-extracted boxes.
The model is pretrained to identify every object referenced in the text. For example, for "a woman in a blue dress standing next to a rose bush," it can learn boxes for the referenced woman, dress, and bush. A referring-expression task instead asks for one box for the object denoted by the entire expression. The authors therefore fine-tuned for five epochs on each of RefCOCO, RefCOCO+, and RefCOCOg, ranking candidate boxes at inference by .
The R101 and ENB3 rows exceed the listed earlier methods on the reported RefCOCO, RefCOCO+, and RefCOCOg splits. However, pre-training image counts vary by row, and some starred earlier methods use detector features overlapping validation or test images. This is not a controlled architecture-only ablation.
-
Visual Question Answering
This model architecture can also be applied to the VQA task but requires some design modifications.
- Model Design
- Query Types: In addition to the 100 detection queries, query types for question types and queries for predicting question types are introduced. In GQA, these question types are defined as REL, OBJ, GLOBAL, CAT, and ATTR.
- Training: Pretraining was conducted for 40 epochs, followed by fine-tuning on the unbalanced GQA split for 125 epochs and final fine-tuning on the balanced split for 10 epochs.
- Loss Strategy: During the first 125 epochs, both detection and QA losses were used, with more weight on QA. The final 10 epochs used QA loss only.
The model utilizes object queries as learned encodings input into the decoder for object detection. During inference, specific parts of the model predict the question type and derive the answer from that part.
- Model Design
-
Performance Comparison
- On GQA test-std, MDETR with a ResNet-101 backbone scores 61.99, above LXMERT's 60.33 and VL-T5's 60.80.
- It also exceeds OSCAR's 61.62 despite OSCAR using more pre-training images; this remains a cross-method, cross-pre-training comparison.
- EfficientNet-B5 raises MDETR to 62.45, while NSM at 63.17 and VinVL at 64.65 are higher in the table. MDETR is not the table's overall best result.
Few-Shot Capabilities?
Inspired by CLIP's zero-shot image classification, the authors asked how to fine-tune pretrained MDETR for detection with little labeled data. Unlike CLIP, MDETR's pre-training does not ensure balanced target classes. More crucially, it lacks negative text queries that name a class but have no matching box. The model can therefore tend to predict a box whenever text is supplied. This does not mean its pre-training lacks boxes aligned with text; those alignments are central to it.
For that reason, the paper does not treat pretrained MDETR as a reliable zero-shot detector and instead evaluates few-shot adaptation. LVIS has about 1,200 categories, many with very few training examples: a long-tailed distribution.
For each positive class, the training example pairs the image with its class name and all boxes for that class. For classes known to be negative for that image, it pairs the name with empty annotations. At inference, MDETR queries each possible class name separately and merges the resulting boxes. The paper reports that this per-class querying takes about 10 seconds per image on a GPU.
The authors fine-tuned MDETR on 1%, 10%, and 100% subsets of LVIS training images. The 1% subset was sampled to include at least one positive and one negative for every category. Baselines were Mask R-CNN trained on all LVIS images and DETR pretrained on MS COCO then fine-tuned on the LVIS subsets. Table 6 reports box AP fixed with per-class prediction caps on a 5,000-image minival set that does not overlap pre-training. With only 1% of LVIS, MDETR scores 11.2 rare-class AP, above DETR's 3.2 after full-data fine-tuning, but below full-data Mask R-CNN's 26.3.
There is also a striking reversal: rare-class AP falls from 20.9 with 10% of the data to 7.4 with 100%. The paper's prose says 7.5, while the imported Table 6 says 7.4; this note follows the table. The authors suggest severe class imbalance as a cause, but that cause was not isolated in this experiment.
Conclusion
One of the most appealing features of MDETR is its "complete differentiability."
This design allows the entire model to be trained end-to-end, fostering tighter model collaboration and potentially improving overall performance and training efficiency. Additionally, its practical performance on multiple datasets has demonstrated impressive results, establishing its position in the multi-modal learning domain.
Furthermore, MDETR's versatility is a major highlight. Not only does it excel in modulated detection, but it has also proven its value in downstream applications such as few-shot detection and visual question answering.
MDETR provides a pathway, with its non-reliance on black-box object detectors potentially inspiring more researchers to create accurate and efficient models.