ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- Publication
- AAAI 2021Conference
- First submitted to arXiv
- arXiv:2006.16934
Cite this paper · BibTeX
The citation identifies the linked source version. It cites the original paper and its authors.
@misc{multimodalityernievil2020,
title = {{ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph}},
author = {Fei Yu and Jiji Tang and Weichong Yin and Yu Sun and Hao Tian and Hua Wu and Haifeng Wang},
year = {2020},
eprint = {2006.16934},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2006.16934}
}
You have probably heard a few stories about the models that followed BERT and could name some differences. For example, Baidu's ERNIE uses phrase- and entity-level masking to learn knowledge during pre-training, and ERNIE 2.0 adds further pre-training tasks. This is different from directly feeding an external entity knowledge graph into the model.
Since ERNIE improved on BERT's way of learning, it is hardly surprising that researchers would try to carry the same idea into other domains. Could an image–text model pay more attention to the words that determine what a scene means?
Problem Definition
The authors focus on a shortcoming of vision-language pre-training models: aligning detailed semantics across images and language. Random subword masking does not explicitly distinguish ordinary words from words describing objects, their attributes or the relationships between them. Yet “a cat on a car” and “a cat under a car” differ in a small phrase and describe different scenes.
The authors define three points they want to address:
- A limitation of current vision-language pre-training: Randomly masking and predicting subwords does not emphasize the words that carry object, attribute and relationship information.
- The importance of detailed semantic alignment: Without emphasizing these distinctions across the two modalities, a model may fail to capture subtle differences in real scenes. The new objective addresses this training bias.
- The data challenge: Vision-language pre-training needs well-aligned image–text pairs, which are hard to obtain at high quality and scale.
Solution
ERNIE-ViL Model Design
Dual-Stream Cross-Modal Network
The dual-stream cross-modal Transformer combines information from the two modalities into a joint vision-language representation:
- Two streams: Text and image regions have their own Transformer paths, so each can process the features of its own modality.
- Text and image inputs: Text arrives as a sequence of subwords, while a detector supplies a sequence of visual region features. The visual Transformer does not directly consume the full two-dimensional RGB image.
- Cross-modal blocks: The paths exchange information through cross-modal attention. They do not remain independent throughout, nor must both finish all their processing before fusion begins.
The aim is easier to picture with an example: if the text says “red ball,” ideally the model will connect those words to the red ball in the image.
Scene Graph Prediction Tasks
A scene graph is a representation of objects, attributes and relationships; the parser producing it is a separate component. ERNIE-ViL parses its scene graphs from the paired captions, not from images. The proposed architecture does not include a GNN.
For “a red apple on a table,” the graph identifies the objects apple and table, the attribute red, and the relationship on. The authors turn those nodes into three prediction tasks:
| Task | Prediction target in the text | Retained context |
|---|---|---|
| Object prediction | Select 30% of the parsed object nodes and predict their corresponding text spans, such as “apple” | Other text and image regions |
| Attribute prediction | Select 30% of object–attribute pairs and predict the attribute text, such as “red” | The associated object word, other text and image regions |
| Relationship prediction | Predict a selected relationship span, such as “on” | Both object words, other text and image regions |
For selected nodes, the paper uses an 80% [MASK], 10% random-token and 10% unchanged-token strategy. The targets are still the original text spans. By practicing these three tasks, the model is encouraged to connect objects, attributes and relations in the description to the visual scene, yielding a richer joint representation.
Look back at Figure 2: the little white cat, blue dress, car and “on top of” are separated in the same way. Mask blue and dress remains; mask on top of and cat and car remain. Image regions can then help select the missing detail. The graph is not a second picture directly fed into the encoder.
Semantic Alignment
Both “on the car” and “under the car” could fit the surrounding grammar. Looking at the image can help eliminate the option inconsistent with the scene. This is the fine-grained alignment the authors want to encourage.
When you see a cat on a car, the useful connection is not just that the caption contains cat and car; it also matters that the cat is on the car. The object, attribute and relationship tasks push the model toward that more precise connection. They encourage visual grounding.
Encoding Methods
Encoding means producing representations the model can process, not necessarily reducing dimensionality.
- Text: WordPiece tokens receive word, segment and position embeddings, with
[CLS]and[SEP]. Splitting depends on the vocabulary:playingis not guaranteed to becomeplayand##ing. - Image: Detector region features are combined with position encodings describing normalized coordinates and area.
[IMG]uses mean-pooled region features and a whole-image position encoding.
Pre-Training Tasks
Pre-training gives the model practice before it is adapted to specific tasks. Here are the tasks and why the author draws attention to each one. The first three are Scene Graph Prediction (SGP) tasks: they predict text tokens, not erased image pixels.
- Object prediction: Objects are central to a visual scene. For an image of an apple on a table, the task hides apple or table in the caption and predicts the word using the remaining text and image regions. It does not hide the apple region in the image.
- Attribute prediction: Color, size and shape tell us more about an object. For “red apple,” the task keeps apple, hides the attribute red in the text, and predicts it using the object, other words and image regions.
- Relationship prediction: In “the apple is on the table,” on describes a relative position. The task keeps apple and table in the text, hides the relationship span and predicts it from the remaining context and image regions.
- Masked language modeling (MLM): Separately from graph-selected SGP targets, MLM randomly selects 15% of text tokens. For “the apple is red,” it could hide red and predict it using the remaining words and image.
- Masked region prediction and image–text matching: Masked Region Prediction (MRP) selects visual region features and predicts them; Image–Text Matching (ITM) classifies whether a picture and caption belong together. These are separate from SGP's text targets. Token and region prediction losses are calculated only for correctly paired image–text examples.
Discussion
ERNIE-ViL reports strong results across several vision-language tasks against its contemporary pre-trained models. These are historical comparisons from the cited paper, not current model rankings.
Advantage of Out-of-Domain Training Data
In Table 3, among methods pre-trained on comparable out-of-domain data such as CC and SBU, ERNIE-ViL leads on the five downstream tasks considered.
- ERNIE-ViL-large scores 66.3% on VCR Q→AR test, versus VLBERT-large’s 59.7%: +6.60 percentage points. VQA test-std rises from 72.22 to 73.96: +1.74 points.
- RefCOCO+ testA/testB rise from 78.57/62.30 to 80.97/64.70, both +2.40 points. This is referring-expression grounding: locating an image region from text.
- Compared with Unicoder-VL-base, ERNIE-ViL-base gains 2.94/0.50 points in Flickr30K image/text retrieval R@1.
- With both out-of-domain and in-domain pre-training data, ERNIE-ViL-large remains ahead of UNITER, OSCAR and VILLA on several tasks. Individual columns still need checking.
Importance of Scene Graph Prediction
With BERT text initialization held fixed, the first two rows show improvements across VCR, VQA, RefCOCO+, image retrieval and text retrieval after adding SGP. For example, VCR Q→AR dev rises from 59.06 to 59.92; RefCOCO+ val rises from 72.81 to 73.50 (+0.69 percentage points), and Flickr30K image-retrieval dev R@1 from 70.74 to 72.96 (+2.22 points). This comparison isolates the objective more clearly than mixing initialization settings.
Keeping SGP but switching text initialization from BERT to ERNIE 2.0 raises VCR Q→AR dev from 59.92 to 61.24. The authors suggest that ERNIE 2.0's other pre-training tasks may supply useful commonsense knowledge; this is an interpretation, not a separately measured causal mechanism. The gains in tasks that require referring expressions and cross-modal retrieval make the scene-graph objective worth examining.
Cloze Tests
The test samples 15,000 Flickr30K image–text pairs and selects 5,000 targets each for objects, attributes and relationships. ACC@1 measures whether the first prediction is correct; ACC@5 whether the correct answer appears among the first five.
Both models use BERT initialization. SGP improves object, attribute and relationship ACC@1 by 1.20, 1.84 and 3.08 percentage points, and overall ACC@1 rises from 49.75 to 51.75. In some illustrated cases the model without SGP misses the target; in another it makes a plausible prediction with lower confidence. Those examples help explain the difference.
Conclusion
ERNIE-ViL's contribution is to use structured language analysis to suggest which cross-modal relationships deserve extra attention during training. That structure may also bring costs and limits. ERNIE-style models can demand substantial computation and storage, with the scene-graph objectives’ additional cost still requiring measurement. Generalization to rare cross-modal scenes is another question worth testing.
The scene-graph tasks invite a practical question. Suppose a caption says “a person standing beside a tree,” but its parser records a mistaken relationship such as sitting. Could that wrong signal affect a downstream task? This is a hypothetical parser error; the paper does not quantify its cost.
This adds uncertainty in practice. Imagine a blurry silhouette or several overlapping objects: the system may struggle to decide whether a shape is a person, a shadow or something else. Such ambiguity could matter where precise recognition or scene interpretation is required. These remain risks to test rather than measured failures from this experiment.
Using a scene graph also seems to imply that the human way of organizing knowledge can guide a model better than what it learns on its own. But is that necessarily a good thing? I would invite you to think about where that guidance helps and where it might constrain the model. That is how we can see both the opportunities and difficulties in the paper's “knowledge enhancement” technique.
Even so, ERNIE-ViL offers a useful direction for cross-modal pre-training by improving the reported fine-grained alignment results. The authors suggest image-derived scene graphs and graph neural networks (GNNs) as future avenues for adding structured knowledge; neither is implemented in this model. The question of how best to use that knowledge remains open for later work.