UNITER: UNiversal Image-TExt Representation Learning
- Publication
- ECCV 2020Conference
- First submitted to arXiv
- arXiv:1909.11740
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityuniter2019,
title = {{UNITER: UNiversal Image-TExt Representation Learning}},
author = {Yen-Chun Chen and Linjie Li and Licheng Yu and Ahmed El Kholy and Faisal Ahmed and Zhe Gan and Yu Cheng and Jingjing Liu},
year = {2019},
eprint = {1909.11740},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/1909.11740}
}
We've seen previous papers evolve from "single-stream models" to "dual-tower models" and "dual-stream models."
Essentially, you can either combine multimodal information in a single computation or process them separately.
If you process them separately, someone will later try to combine them and surpass you. If you combine them, someone will eventually separate them and surpass you.
We just wanted to casually read some papers, but we've stumbled upon some life lessons.
Now, let's get back to UNITER.
In UNITER’s period, MCB, BAN, DFAF, SCAN and MAttNet each specialized in particular tasks. Their task-specific fusion designs were not easy to transfer. UNITER asks whether shared image–text representations can be learned first and then adapted to different tasks.
Problem Definition
The question goes beyond single versus dual streams. The authors focus on cross-modal pre-training objectives, which clues to retain when masking inputs, and fine-grained alignment between words and image regions.
The “universal” ambition is evaluated across six task families and nine datasets.
Can we learn a genuinely universal image–text representation for every V+L task? UNITER tests part of that large question. Its evaluation covers nine datasets.
Solution
UNITER Model Design
UNITER is a single-stream Transformer. Image and text inputs first become embeddings; their tokens then enter the same Transformer. The Image/Text Embedders are not two independent deep Transformer stacks.
- Image: Faster R-CNN supplies region features. A seven-dimensional vector describes normalized coordinates, width, height and area. Visual and location features are projected separately, summed and layer-normalized.
- Text: WordPiece token embeddings receive position information; both modalities also receive modality embeddings.
- Interaction: Joint self-attention connects textual tokens and image regions to produce contextualized representations.
The key masking choice is conditional masking: keep the image intact during MLM and the text intact during MRM. Masking both the word “dog” and its image region could remove the corresponding clues from both sides. Masking one modality reduces that problem without guaranteeing a unique region match for every word.
Pre-Training Strategy
Let's go through the four pre-training objectives one by one. They have distinct roles:
| Task | Input and learning target |
|---|---|
| MLM | Select 15% of text tokens, replacing 80% with [MASK], 10% with random tokens and leaving 10% unchanged; predict the originals using the remaining text and full image. |
| ITM | Classify whether image and sentence match, using the fused [CLS] representation and positive/negative pairs. |
| WRA | Encourage fine-grained token–region alignment by minimizing the optimal transport (OT) distance. |
| MRM | Select about 15% of image regions, mask their features, and predict targets using other regions and the complete text. |
A lawn photograph can help recover “lawn” in “The puppy is playing in the [MASK].” This is an available cross-modal clue.
ITM negatives replace the image or sentence. A fused [CLS] representation passes through a classifier trained with binary cross-entropy. Its score is an estimate for the matching decision, not a calibrated percentage of semantic similarity.
Consider “a beach at sunset.” The caption with a sunset beach is a positive pair; switch in a mountain photo and it becomes a negative. The fused [CLS] representation goes to a classifier that learns whether the pair matches. It does not draw a beach or measure exactly how similar two photographs are.
What does WRA’s OT do?
Treat words and regions as two sets of feature points carrying probability mass. A transport plan assigns mass between them while respecting marginal distributions and reducing total matching cost. Its total mass is one. After approximating the optimal plan, UNITER uses the optimal transport distance—the plan-weighted sum of word–region costs—as its scalar WRA loss to update the model. The plan itself supplies the weights; it is not the loss.
Unregularized discrete OT admits sparse optimal solutions, but finite approximate iterations need not produce exact sparsity. Nor does OT guarantee semantically correct one-to-one matches. The paper uses IPOT for approximation; matching “red apple” to a red apple is an illustrative goal, not an extra human alignment label.
MRM has three variants:
- MRFR regresses detector region features, not pixels.
- MRC uses the detector’s most confident category as a hard target. The label granularity follows the detector’s vocabulary, such as the category “bird.”
- MRC-kl matches the detector’s full category distribution through KL divergence, retaining soft-label information. These targets inherit the detector’s vocabulary and errors.
Consider a flock of birds. If one bird region is masked, MRFR guesses its detector feature vector, MRC guesses the detector's top category, and MRC-kl learns its full category distribution. None paints the missing bird pixels back in; none guarantees that the model can name a bird species.
Pre-Training Datasets
COCO + Visual Genome (VG) form the in-domain group; Conceptual Captions (CC) + SBU Captions form the out-of-domain group. These labels describe distribution relative to downstream evaluation, not data quality.
The processed COCO/VG group provides about 5.6 million training image–text pairs and 131K internal validation pairs. Pair counts are not unique-image counts. Downstream evaluation images are excluded, and URL matching removes identified overlap with Flickr30K.
Discussion
Performance on Pre-Training Tasks
These ablations measure downstream performance after pre-training on VQA, NLVR2, Flickr30K and RefCOCO+. Meta-Sum adds the listed scores. It helps compare identical evaluation settings, but is not a universally meaningful quality unit.
- L1 has no pre-training; text-BERT initialization in L2 already raises Meta-Sum from 314.34 to 346.24.
- With a single image–text objective, ITM-only (L4) and MLM-only (L5) improve all the listed downstream scores over L2. MRFR-only (L3) exceeds L2 only on NLVR2.
- Combining MLM and ITM (L6) raises Meta-Sum above either alone, although its RefCOCO+ score is below MLM-only L5. MRM variants differ: MRC-kl generally beats hard-label MRC, and MRFR complements it.
- Among the in-domain image–text rows L3–L12, adding WRA gives L11 the highest Meta-Sum: 399.97 in L10 rises to 400.93 in L11. VQA rises from 71.92 to 72.47 and RefCOCO+ from 74.52 to 74.80, while Flickr image retrieval and NLVR2 dip slightly. Using both data groups in L14 raises Meta-Sum further to 405.24; L11 is not the highest row in the whole table.
- Keeping one modality intact during masking matters here. With the same objectives but simultaneous random masking of both modalities, L12 falls to 396.51 from L10's 399.97.
- Out-of-domain-only L13 scores 396.91, below in-domain L11's 400.93; combining both data groups in L14 reaches 405.24. Data quantity and distribution similarity are separate considerations.
Performance on Downstream Tasks
UNITER-large leads many rows in the paper's contemporary comparison, not today's rankings. It reaches 73.82 on VQA test-dev and 79.12 on NLVR2 dev; its Flickr zero-shot text-retrieval R@10, however, is 97.70, slightly below UNITER-base's 98.00. In v3 Table 3, UNITER-base scores 58.20 on VCR Q→AR, 3.2 percentage points above the task-specific 55.00 baseline but below VL-BERT-large's 59.70; UNITER-large reaches 62.80. On VQA test-dev, UNITER-base scores 72.70, above ViLBERT's 70.55 and VisualBERT's 70.80, but only 0.28 points above LXMERT's 72.42. The models' pre-training data and tasks differ, so these are table comparisons rather than controlled tests of stream architecture.
A single stream is competitive, with comparisons affected by differences in data, objectives and model sizes. The quoted 86M/183M/221M parameter comparison excludes word embeddings, so it is not a full deployment-size comparison.
Two task adaptations are useful to retain:
- VCR: general image–text pre-training, then a second pre-training stage on VCR training data, followed by task fine-tuning. Evaluation answers are not training inputs here.
- NLVR2: a sentence refers to two images, requiring additional inter-image interaction. In the UNITER-base adaptation study, Triplet scores 73.03 on dev, Pair 75.85, and Pair-biattn 77.18. The added bidirectional attention helps the two images interact; it is a task adaptation, not a conversion of all UNITER pre-training into a dual-stream design.
What Did the Model Learn?
Attention maps show connection patterns.
| Pattern | Observable behavior |
|---|---|
| Vertical | Many positions attend to the same token, possibly [CLS] or [SEP]. |
| Diagonal | Attention concentrates on the same or nearby positions. |
| Vertical + diagonal | Both shapes occur together. |
| Block | Stronger attention within a modality. |
| Heterogeneous | No simple pattern of the above forms. |
| Reversed block | Stronger cross-modal attention. |
These descriptions start an analysis. Establishing a head’s causal role requires further interventions or ablations.
Conclusion
Looking back, we can see how much interest the early vision-language researchers had in finding a representation that would carry across tasks. Single- versus dual-stream debates will continue, but UNITER reminds us that what to mask, how to align and which data to use also shape results.
The authors examined their four pre-training objectives through ablations and trained with both in-domain and out-of-domain data. UNITER was an important step in general image–text representation learning at the time. It also gives us a chance to reflect on the field's path and on further work in image–sentence interaction and more efficient pre-training.