UNITER: Learning UNiversal Image-TExt Representations

By Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu. 2019

477 citations recorded by the source corpus.

Joint image-text embedding is the bedrock for most Vision-and-Language (V+L) tasks, where multimodality inputs are jointly processed for visual and textual understanding. In this paper, we introduce UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets (COCO, Visual Genome, Conceptual Captions, and SBU Captions), which can power heterogeneous downstream V+L tasks with joint multimodal embeddings. We design three pre-training tasks: Masked Language Modeling (MLM), Image-Text Matching (ITM), and Masked Region Modeling (MRM, with three variants). Different from concurrent work on multimodal pre-training that apply joint random masking to both modalities, we use conditioned masking on pre-training tasks (i.e., masked language/region modeling is conditioned on full observation of image/text). Comprehensive analysis shows that conditioned masking yields better performance than unconditioned masking. We also conduct a thorough ablation study to find an optimal setting for the combination of pre-training tasks. Extensive experiments show that UNITER achieves new state of the art across six V+L tasks (over nine datasets), including Visual Question Answering, Image-Text Retrieval, Referring Expression Comprehension, Visual Commonsense Reasoning, Visual Entailment, and NLVR2.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Language Models are Few-Shot Learners (2020, background): “Recent work [RWC + 19] suggested this ﬁnal step may not be necessary.”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “For the former, we reﬁne the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “The original UNITER-base (12 layers) and UNITER-large (24 layers) models take 200k and 500k steps for pre-training, respectively.”
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “2 - - - - - UNITER [5] B - - - - - - 63.”
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “UNITER[21] added partial in-domain datasets (VG and MSCOCO) during pre-training stage, thus getting the highest zero-shot results.”
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “UNITER add about 1 million image-caption pair besides the Conceptual Captions and in-domain MSCOCO Caption and Visual Genome Dense Captions (Krishna et al. 2017) data.”
Perceiver: General Perception with Iterative Attention (2021, methodology): “A variety of strategies have been proposed to reduce the size of the input to the Transformer so it can be used on domains that are otherwise too large, including subsampling the input (Chen et al., 2020a) or by ﬁrst preprocessing the input using convolutions (e.g. Wu et al. 2020).”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “As a result, the highest performing models on vision-and-language tasks, including Visual Commonsense Reasoning (VCR) (where Figure 1’s scene originates from), learn about the visual world exclusively through manuallyannotated bounding boxes, along with image captions [95, 17, 58, 63, 107, 30].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Based directly upon BERT, V+L pre-training research has largely been focused on V+L understanding [52, 46, 21, 72, 3, 69, 45, 53], with classiﬁcation or regression tasks that do not involve generation.”
ActBERT: Learning Global-Local Video-Text Representations (2020, methodology): “In the past year, many works extended BERT to model cross-modal data [21, 32, 34, 5, 19, 33].”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “Satellited around BERT, there are also studies on model compression (Sun et al., 2019c; Jiao et al., 2019; Shen et al., 2019) and extension from understanding to generation (Chen et al., 2019a; Clinchant et al., 2019; Wang and Cho, 2019).”
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020, background): “…such as ViLBERT (Lu et al., 2019) and LXMERT (Tan and Bansal, 2019), or a single-stream architecture to encode both text and image together, such as B2T2 (Alberti et al., 2019), Unicoder-VL (Li et al., 2019a), VisualBERT (Li et al., 2019b), VL-BERT (Su et al., 2019), and UNITER (Chen et al., 2019).”
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline (2019, background): “Recent work has extended this to vision and language tasks [2, 45–50], leading to compelling results in Visual Question Answering [4], Commonsense Reasoning [51], Natural Language Visual Reasoning [52], Entailment [53], Image-Text Retrieval [54, 55], Referring Expressions [56], and Vision-Language N”
Grounded Language-Image Pre-training (2021, background): “In visionlanguage literature [7, 19, 27, 28, 30, 36, 48, 50, 65], deep fusion of visual and language features is necessary to learn a performant phrase grounding model.”
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers (2020, methodology): “To learn a universal representation for most vision and language tasks, we ﬁrst pre-train our model with image-sentence pair dataset similar to other cross-modality pre-training researches [6,18,19,21,22,29,33,41].”