UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning

By Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Hua Wu, Haifeng Wang. 2020

425 citations recorded by the source corpus.

Existed pre-training methods either focus on single-modal tasks or multi-modal tasks, and cannot effectively adapt to each other. They can only utilize single-modal data (i.e., text or image) or limited multi-modal data (i.e., image-text pairs). In this work, we propose a UNIfied-MOdal pre-training architecture, namely UNIMO, which can effectively adapt to both single-modal and multi-modal understanding and generation tasks. Large scale of free text corpus and image collections are utilized to improve the capability of visual and textual understanding, and cross-modal contrastive learning (CMCL) is leveraged to align the textual and visual information into a unified semantic space, over a corpus of image-text pairs augmented with related images and texts. With the help of rich non-paired single-modal data, our model is able to learn more generalizable representations, by allowing textual knowledge and visual knowledge to enhance each other in the unified semantic space. The experimental results show that UNIMO greatly improves the performance of several single-modal and multi-modal downstream tasks. Our code and pre-trained models are public at https://github.com/PaddlePaddle/Research/tree/master/NLP/UNIMO.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

All in One: Exploring Unified Video-Language Pre-Training (2022, background): “In image-text pre-training, the early work Unimo [22] solves both understanding and generation tasks with cross-modal contrastive learning.”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “In this section, we compare BLIP to existing VLP methods on a wide range of vision-language downstream tasks2.”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Florence outperforms in 7/11 tasks over 5-shot fine tuning, and outperforms full-set fine-tuning on the “Packages” 2https://public.roboflow.com/object-detection 3We refer to (Li et al., 2021b) for details.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “(2) Another line of work (Li et al., 2021a; Kim et al., 2021) enforces the token-wise or patch-wise representations from both modalities into the same space and models these ﬁner-grained interactions via cross-attention (Li et al., 2021a) or self-attention (Kim et al., 2021).”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “The second category models the interaction of images and text using a deep fusion encoder with cross-modal attention [43, 30, 41, 24, 51, 3, 26, 25, 14, 49, 16, 17, 20, 23, 46].”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, methodology): “…detection for image region feature regression or tagging as part of the pre-training objectives (Tan & Bansal, 2019; Su et al., 2020; Li et al., 2019; Chen et al., 2020b; Gan et al., 2020; Li et al., 2020; Yu et al., 2021; Li et al., 2021; Zhang et al., 2021; Hu et al., 2021; Cho et al., 2021).”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “Recently, the image-text contrastive loss has also been added in Yu et al. (2022); Li et al. (2021a); Wang et al. (2021a).”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “Multimodal Pretraining Multimodal pretraining has been developing rapidly [38, 13, 39, 40, 14, 41, 42, 43, 44, 15, 16, 17, 45, 46, 47].”
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022, background): “…language modeling (Tsimpoukelli et al., 2021; Alayrac et al., 2022), masked language modeling (Tan and Bansal, 2019; Lu et al., 2019; Li et al., 2021b), preﬁx language modeling (Wang et al., 2022), image-text matching (Li et al., 2019; Chen et al., 2020; Li et al., 2020) and…”
A Survey of Transformers (2021, background): “, visual question answering [55, 75, 77, 125], visual commonsense reasoning [75, 125], caption generation [22, 81, 128], speech-to-text translation [46] and text-to-image generation [29, 81, 107].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “46 UNIMO [163] arXiv-2020 image-text Trans CS Adapt to single-, multi-modal understanding and generation tasks effectively URL”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, neutral): “[220], [221], [222], [223], [224], [225].”
Multi-Modal Knowledge Graph Construction and Application: A Survey (2022, methodology): “Models have proven their effectiveness in improving many downstream tasks [48], [49], [50], [51], [52].”
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models (2021, neutral): “…Auto-regressive language modeling based VL-PTMs model image and text tokens with Transformer decoders auto-regressively (Ramesh et al., 2021; Wang et al., 2021); (3) Contrastive learning based VL-PTMs are pre-trained to holistically match image-text pairs (Radford et al., 2021; Li et al., 2021b).”
Vision-Language Pre-Training with Triple Contrastive Learning (2022, background): “Among them, OSCAR [28], UNIMO [27], VILLA [16], and UNITER [8] use an object detector (e.g., Faster R-CNN [42]) to capture vision features first, then a multi-layer transformer [45] is applied to the concatenation of the extracted vision features and text features to learn joint embeddings.”