Back to paper library
arXiv 2020ACL 2021Multimodal learningPaper notes

UNIMO

Intertwined Clues

Original paper

UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

Paper authors

Wei Li · Can Gao · Guocheng Niu …

Show all 8 authors

Wei Li · Can Gao · Guocheng Niu · Xinyan Xiao · Hao Liu · Jiachen Liu · Hua Wu · Haifeng Wang

Publication
ACL 2021Conference
First submitted to arXiv
arXiv:2012.15409
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{multimodalityunimo2020,
  title = {{UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning}},
  author = {Wei Li and Can Gao and Guocheng Niu and Xinyan Xiao and Hao Liu and Jiachen Liu and Hua Wu and Haifeng Wang},
  year = {2020},
  eprint = {2012.15409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.15409}
}
On this page

Hasn't it been a while since Chinese class? What is cuòzōng (錯綜)—do you still remember? This may seem like an odd turn, but let's review it first.

Rewriting as a Rhetorical Analogy

  1. Rearranging word order: Change the order of the original sentence to alter its rhythm or emphasis. The classical phrase 「陟罰臧否」 becomes 「陟臧罰否」.

  2. Replacing words: Substitute words to change the expression. 「南山烈烈,飄風發發」 becomes 「南山律律,飄風弗弗」.

  3. Expanding or contracting a sentence: Change its length to create a different rhythm. 「於山見終南嵩華之高,於水見黃河之大且深」 becomes 「於山見高,於水見大且深」—the shorter version loses the named mountains and river.

  4. Changing the sentence form: Turn a statement into a question, or change its tone. 「徐公何能及君也?」 becomes 「徐公能及君也嗎?」.

Cuòzōng plays with language structure, order, length and form.

Back to the paper. UNIMO uses something like word substitution while training a multimodal model, generating more positive and negative image–text examples. Is that the whole trick?

The paper also uses back-translation and image/text retrieval, so word substitution is only one part of the method. Still, the rhetoric lesson gives us a question to carry into the experiments: which kinds of rewriting or retrieval actually contribute, and how much?

Problem Definition

Looking back at earlier pretraining work, the authors separate unimodal and multimodal approaches and identify several limitations:

  1. Challenges of unimodal learning

    A model focused on text or images alone cannot directly combine information from both in a multimodal task. That limits what its representations can do without additional cross-modal training.

  2. Limited paired multimodal data

    Methods such as ViLBERT, VisualBERT and UNITER learn from image–text pairs, which are scarcer than standalone text or images. The authors argue that these models may also transfer less well to unimodal tasks.

  3. Aligning multimodal semantic spaces

    Image and text representations must be aligned before their relationships can be used effectively across modalities. Establishing a shared semantic space that transfers beyond the training pairs remains difficult.

  4. Learning cross-modal representations efficiently

    Existing approaches often rely on image–text matching and masked language modeling over limited paired data. The paper asks how to learn more general visual and language representations from the much larger unpaired collections as well.

  5. Cross-modal understanding and generation

    Visual and language knowledge might help each other in both understanding and generation tasks. Here, “generation” means text generation, not image synthesis.

UNIMO's answer is to train on text-only, image-only and paired image–text data in one framework. These are the paper's motivations, not a universal claim that every unimodal or earlier multimodal model fails to transfer.

Solution

UNIMO Model Design

UNIMO is a cross-modal pretraining method designed to capture and learn modality-invariant information at the semantic level across different modalities. The core concept of the model is derived from how humans perceive the world through multiple senses.

The main design principles are:

  1. Multi-Modal Learning

    • UNIMO learns from different types of data (e.g., images, text, and image-text pairs) to capture modality-invariant information at the semantic level, achieving more robust and general representations for both text and visual inputs.
  2. Adopting Transformer Architecture

    • UNIMO uses a multi-layer self-attention Transformer structure to learn unified semantic representations of text and visual data.
    • Text input is first segmented into subword sequences using Byte Pair Encoding (BPE) and contextualized token representations are learned using the self-attention mechanism.
    • For image input, Faster R-CNN is used to detect salient image regions and extract visual features for each region.
  3. Cross-Modal Contrastive Learning (CMCL)

    • CMCL aims to align and unify text and visual representations at different levels into a single semantic space.
    • The basic idea is to bring paired image and text representations closer together in the representation space while pushing unpaired representations further apart.
    • Positive image-text pairs X+X^+ and negative image-text pairs X−X^- are created using a series of text rewriting techniques, and relevant images XIX_I and text XTX_T are retrieved from single-modal data.
    • The same unified-modal Transformer encodes paired image–text data together. Retrieved unimodal images and text, which are only weakly related to the pair, are encoded separately; their representations then enter the contrastive loss.
  4. Joint Learning of Single and Multi-Modalities

    • UNIMO is pretrained on large image sets, text corpora, and image-text pairs, learning generalized visual and textual representations through masked prediction and unifying them into the same semantic space.
    • Visual regions are trained with masked-feature regression and region classification. For text, different attention masks support both bidirectional prediction and autoregressive Seq2Seq generation. The experiments ask whether these visual and language signals can help one another in the shared space.

CMCL

CMCL increases scores for related examples relative to unrelated ones:

In the formula:

  • Although the paper also calls d(V,W)d(V,W) a distance, its exp⁡(d/τ)\exp(d/\tau) term requires a higher-is-more-similar score for positive pairs. A conventional smaller-is-better distance would need a transformation such as negation here.
  • τ>0\tau>0 controls the sharpness of the resulting distribution.
  • The numerator contains positive image–text pairs and related unimodal images and texts. To score a retrieved image, the method pairs it with the original text; to score retrieved text, it pairs it with the original image.
  • The denominator contains those examples and negative image–text pairs, images and texts; it is not a sum over negatives alone.
  • EV,W\mathbb{E}_{V,W} averages over sampled training image–text pairs.

Minimizing the negative log increases the share of score mass assigned to positives. Raising a negative score increases the denominator and worsens the loss; it is not the case that “more distant negatives have larger positive exponentials.”

The positive terms increase when their similarity scores increase; lowering a negative score reduces its exponential term in the denominator. That is how minimizing the negative log favors related samples over unrelated ones.

Multi-Granularity Semantic Alignment

Positive/negative labels below are construction assumptions: back-translation, similarity retrieval and random replacement can all introduce mistakes.

  • Text Rewriting

    This process plays a crucial role in enhancing multi-granularity semantic alignment between images and texts. The titles or descriptions of images are rewritten at different levels (sentence level, phrase level, and word level) to provide a rich learning environment for the model, helping it align the semantics of images and languages more precisely.

    1. Sentence-Level Rewriting

      • Positive Sample Generation: Using back-translation techniques. Each image title is first translated into another language and then back to the original language. This method allows the model to obtain multiple semantically similar but syntactically and lexically different titles for an image as positive samples.
      • Negative Sample Generation: Using TF-IDF similarity-based retrieval to find titles of other images that are very similar to the original image title but do not accurately describe the original image, thus serving as hard negative samples.
    2. Phrase-Level and Word-Level Rewriting

      • The image captions are first parsed into scene graphs of objects, attributes and relationships. The method replaces an object, attribute or relationship node with a similar entry from a corresponding vocabulary, then selects a fluent rewritten caption as a word- or phrase-level hard negative for the original image.

    Unlike previous methods that often rely on random sampling of negative samples, text rewriting can generate numerous hard negative samples that are semantically close to positive samples but contain subtle differences, thus challenging the model during the learning process.

    These text rewriting strategies help the model learn different levels of semantic alignment between images and texts. By contrasting positive and negative samples, the model learns to align semantically similar images and texts while distinguishing semantically mismatched images and texts. In this way, text rewriting enhances the model's semantic understanding not only at the sentence level but also at finer semantic levels (e.g., words and phrases).

  • Image/Text Retrieval

    The main goal is to integrate more single-modal information into the cross-modal learning process to enhance the model's learning effect in semantic alignment.

    1. Augmenting Single-Modal Information

      Each image-text pair is further augmented by retrieving related images and texts from single-modal data (data containing only images or only text).

      • Image Augmentation: Build TF-IDF features from detected object labels and retrieve images with similar objects.
      • Text Augmentation: Use TF-IDF to retrieve candidate sentences, then rerank them by the similarity of their BERT representations to the original caption.
    2. Encoding Augmented Information

      The augmented single-modal image and text information are further encoded to calculate cross-modal contrastive loss.

      • Single-Modal Transformer Encoding: The augmented images and texts are individually encoded by a unified modality Transformer. This implies that the model learns to map these single-modal data into a shared semantic space to facilitate the correlation between images and texts.
      • Calculating Cross-Modal Contrastive Loss: In this semantic space, the representations of augmented single-modal information are extracted to compute the contrastive loss as previously mentioned.
    3. Enhancing Cross-Modal Learning with Single-Modal Information

      These augmented images and texts provide rich contextual information, playing a critical role in cross-modal learning by helping the model understand and establish stronger semantic alignment between visual and language modalities.

      This process enhances the model's semantic understanding and alignment capabilities in the cross-modal learning process by augmenting the original image-text pairs with additional retrieved single-modal information, improving the model's learning efficiency and performance.

Experiment Design

In this section, the authors detail the experiment design for the multi-modal pretraining model UNIMO, including pretraining and fine-tuning setups.

  1. Pretraining Dataset and Implementation Details

    • Pretraining Datasets

      • Text Corpus: Using BookWiki and OpenWebText as primary text sources.
      • Image Collections: Including subsets of OpenImages and COCO unlabel, which are pure image data without text descriptions.
      • Image-Text Pairs: Integrating four multi-modal datasets: COCO, Visual Genome, Conceptual Captions, and SBU Captions.
    • Implementation Details

      • Model Sizes: UNIMO offers two model sizes – UNIMO-base (12-layer Transformer) and UNIMO-large (24-layer Transformer).
      • Pretraining: The models are initialized from RoBERTa-base and RoBERTa-large and trained for at least 500K steps.
      • Optimization and Hardware: Using Adam optimizer, float16 mixed-precision training, and training on Nvidia Tesla V100 32GB GPUs.
  2. Fine-Tuning Tasks and Experiment Settings

    • Single-Modal Tasks

      • Generation Tasks: Including conversational question answering (CoQA), question generation (SQuAD 1.1), abstractive summarization (CNN/DailyMail), and sentence compression (Gigaword).
      • Understanding Tasks: Covering sentiment classification (SST-2), natural language inference (MNLI), linguistic acceptability analysis (CoLA), and semantic similarity analysis (STS-B).
    • Multi-Modal Tasks

      • Including visual question answering (VQA v2.0), image captioning (Microsoft COCO Captions), visual language inference (SNLI-VE), and image-text retrieval (Flickr30k).

In UNIMO's training process, Faster R-CNN pretrained on the VisualGenome dataset is used to extract visual features of salient regions, with clear standards for region category detection and selection. Additionally, positive and hard negative samples are carefully established and selected in the CMCL setting.

Discussion

Performance on Multi-Modal Tasks

UNIMO shows competitive performance in multi-modal tasks, particularly when compared to leading models such as ViLBERT, VLP, UNITER, and Oscar.

Notably, UNIMO-large surpasses previous leading models in several key test items, especially in image retrieval and text retrieval tasks, improving the R@1 indicator by 1.34 and 1.3 points compared to ERNIE-ViL-large, respectively. This indicates significant improvements in certain image-text retrieval tasks.

UNIMO-large scores 39.6 BLEU-4 versus Oscar-large’s 37.4, but its CIDEr is slightly lower: 127.7 versus 127.8. Improvement in one metric is not dominance across all generation metrics. These are historical results; the cited ERNIE-ViL numbers also differ from later revisions.

Performance on Single-Modal Tasks

UNIMO exhibits robust adaptability in single-modal tasks. Compared with pretrained language models such as BERT, RoBERTa, XLNet and UniLM, it is comparable or slightly ahead on many of the listed language understanding and generation tasks. UNIMO-large is still below RoBERTa-large on MNLI and CoQA. The UniLM comparison shows that a framework trained for both modalities can remain competitive on language-only tasks too.

The results support the framework’s usefulness but do not isolate architecture as the sole cause. The w/o single-modal variant removes unimodal learning and replaces CMCL with ITM, changing more than one factor. RoBERTa generation baselines use the UNIMO architecture initialized with RoBERTa weights, rather than directly generating with an unchanged bidirectional encoder.

Whose Contribution?

In the realm of multi-modal learning that integrates text and visual information, the interaction of these two data forms and their respective impacts on the learning process have always been a hot research topic.

The authors specifically explored how text and visual data mutually reinforce each other within the UNIMO model framework:

  • Impact of Text: Adding Context to Visuals

    w/o texts removes language learning on additional text corpora, not all captions from image–text pairs. Flickr30K image/text retrieval R@1 falls from 74.66/89.70 to 72.04/85.80; VQA test-dev changes only from 73.79 to 73.77, or 0.02 points. Text helps here, but the effect sizes vary.

  • Contribution of Visuals: Enriching and Enhancing Language Representations

    Removing images and paired data lowers most language-understanding and listed generation scores. MNLI m/mm is an exception, rising from 86.8/86.7 to 87.4/86.8. This supports benefits on some tasks.

    The interplay and complementarity of text and visual information in multi-modal learning open a new research perspective and reveal potential future development directions. According to this study's findings, they form a "win-win" situation in the unified semantic space, enhancing overall learning outcomes through their respective contributions. Text data provides context and semantic frameworks for visual elements, while visual data injects vivid concrete content and more real-world scene information into language representations.

Conclusion

In the discussion above, multi-granularity semantic alignment sounds like a key part of how text and visual information meet at different levels. Why, then, is there no dedicated experiment isolating it? Does that mean its role in UNIMO was smaller than expected? We cannot draw that conclusion from an absent component-wise ablation, but I still want to know.

For example, we could remove one fine-grained rewriting route while holding data volume, model and training budget fixed. How much would each cross-modal task change? Do particular tasks need finer alignment, or might some still work acceptably without it? The paper and appendix provide data-source ablations and examples, but they do not give each rewriting or retrieval component its own measured contribution.

Back-translation could change meaning; a replacement could fail to create a mismatch; related images might legitimately share a caption. These are possible sources of construction noise, not measured error rates in this paper. I hope those questions can inspire more careful experiments on multi-granularity alignment and help us decide which parts of this idea actually earn their place.