UNITER: UNiversal Image-TExt Representation Learning

By Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu. 2019

2,659 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classiﬁcation (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or uniﬁed text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “Early-fusion architectures were shown to outperform the late-fusion ones for the task [15]; therefore, we focus on three early-fusion pretrained models, namely LXMERT [34], UNITER [3] and Oscar [20].”
Training data-efficient image transformers & distillation through attention (2020, background): “Nevertheless hybrid architectures that combine convnets and transformers, including the self-attention mechanism, have recently exhibited competitive results in image classification [53], detection [6, 26], video processing [42, 50], unsupervised object discovery [33], and unified text-vision tasks ”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Vision-Language Modelling Pre-training methods that have recently moved from raw text to multi-modal data (e.g., image-text) have revolutionized numerous multimodal tasks (e.g., image-text matching [9, 27, 34], image captioning [44], Visual Question Answering [9,20,33,34]).”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “…V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Method Pre-train COCO (5K test set) Flickr30K (1K test set) # Images TR IR TR IR R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 UNITER (Chen et al., 2020) 4M 65.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “10 In fact, our Reserve-L requires one-fifth the FLOPs of detector-based systems, like UNITER-Large [20] (Appendix B.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “ViLBERT [32], UNITER [6] and LXMERT [48] use the faster R-CNN [43] object detector to extract the bottom-up [1] features from the image and provide them with text features to a cross-modality transformer to solve visual question answering and image-text alignment tasks.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “Encoder VL models are often also pretrained with auxiliary objectives or custom cross-modal loss functions, such as the Masked Region Modeling, Image-Text Matching and Word-Region Alignment of UNITER [10], or the contrastive loss of OSCAR [29].”
FLAVA: A Foundational Language And Vision Alignment Model (2021, methodology): “Within the fusion encoder category, a further distinction can be made as to whether the model uses a single transformer for early and unconstrained fusion between modalities (e.g., VisualBERT, UNITER, VLBERT, OSCAR [16, 63, 65, 101, 120]) or allows cross-attention only in specific co-attention trans”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “The most widely used objectives are image-text retrieval [2, 5, 43, 55, 63, 73, 74, 75], masked language modeling with image clues [2, 23, 43, 55, 72, 73, 74, 75], and masked region modeling [15, 55, 72, 73, 75].”