Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

By Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, Ming Zhou. 2019

975 citations recorded by the source corpus.

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM (Lample and Conneau 2019) and Unicoder (Huang et al. 2019), both visual and linguistic contents are fed into a multi-layer Transformer (Vaswani et al. 2017) for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling(MLM), Masked Object Classification(MOC) and Visual-linguistic Matching(VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, neutral): “Finally, CLIP is related to a recent burst of activity on learning joint models of vision and language (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2019; Li et al., 2020b; Yu et al., 2020).”
UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Concurrent pre-trained models include: ViLBERT, LXMERT, Unicoder-VL, VisualBERT and VLBERT.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “Unicoder [47] focuses exclusively on image caption retrieval tasks with online hardest negative mining.”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Pre-trained on the same out-of-domain datasets (CC, SBU), ERNIE-ViL acquires signiﬁcant improvements on VCR, Image Retrieval and Text Retrieval compared to Unicoder-VL [5].”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “During the finetuning stage, ITR is formulated as a ranking problem, with triplet loss used for modeling training and hard negatives applied to boost performance [6].”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “Following [19], we report the top-K retrieval results on both the 1K and 5K COCO test sets.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “For example, most recent cross-modal pre-trained models [16, 17, 18, 19, 20, 21] use following 2 datasets in pre-training: The Conceptual Captions (CC) dataset[2], which contains 3M images with descriptions harvested from the Alt-text HTML attribute of the web pages, and SBU Captions[3], which consi”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, methodology): “There have been a number of recent developments demonstrating the feasibility of incorporating external knowledge into Question Answering models (Wang et al., 2017b; Li et al., 2020b; Marino et al., 2021; Wu et al., 2021; Garderes et al., 2020).”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Since the birth of ViLBERT [30] and LXMERT [40], we have witnessed a boom of methods for vision-language pre-training [6,7,13,22,26, 37, 45, 46].”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The image can be encoded as object features [5,6,13,18,27,29,30,30,32,39,42,47,53, 54] through a Faster RCNN [36], as grid features [20, 22] from a convolutional neural network, or as patch features [25] from a linear projection on raw pixels.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “(1) One line of work (Chen et al., 2020; Li et al., 2020b; Dong et al., 2021; Li et al., 2021b; Zhang et al., 2021; Zhan et al., 2021) uses a pre-trained object detector to extract region-of-interest (ROI) features from images, and then fuses it with the paired text through a VLP model.”
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “The first category focuses on modelling the interactions between image and text features with transformerbased multimodal encoders [10, 11, 12, 13, 1, 14, 15, 2, 3, 16, 8, 17, 18].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “While most performant visionand-language models pre-extract features from a (supervised) object detector [108, 69, 75, 22, 68], for the sake of pre-training efficiency we use a grid-based hybrid ResNet/Vision Transformer.”