Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

By Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei and 2 more. 2020

2,259 citations recorded by the source corpus.

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, neutral): “Finally, CLIP is related to a recent burst of activity on learning joint models of vision and language (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2019; Li et al., 2020b; Yu et al., 2020).”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “For the former, we reﬁne the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “More recent studies leverage multi-task learning [36] to enhance finetuning and use detected image tags [32] to further enhance pre-training.”
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Vision-Language Modelling Pre-training methods that have recently moved from raw text to multi-modal data (e.g., image-text) have revolutionized numerous multimodal tasks (e.g., image-text matching [9, 27, 34], image captioning [44], Visual Question Answering [9,20,33,34]).”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “…V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “For example, pre-training with multi-modal data [27, 52], web hypertext [2], or web instruction-action mapping [38] could help agents better understand and leverage rich semantics of webpage content, actions, and instructions.”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Due to the prohibitive expense of acquiring human-annotated texts, most methods (Chen et al., 2020; Li et al., 2020; 2021a; Wang et al., 2021; Radford et al., 2021) use image and alt-text pairs crawled from the web (Sharma et al.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “SOTA model SimVLM [55] PICa [58] CFR [33] Pythia [46] SimVLM SimVLM VIVO [22] SimVLM OSCAR [29]”
FLAVA: A Foundational Language And Vision Alignment Model (2021, background): “Within the fusion encoder category, a further distinction can be made as to whether the model uses a single transformer for early and unconstrained fusion between modalities (e.g., VisualBERT, UNITER, VLBERT, OSCAR [16, 63, 65, 101, 120]) or allows cross-attention only in specific co-attention trans”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Prominent examples include UNITER [6], VLBERT [37], OSCAR [29], UNIMO [28], and VinVL [48].”