ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
By Jiasen Lu, Dhruv Batra, Devi Parikh, Stefan Lee. 2019
4,585 citations recorded by the source corpus.
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring expressions, and caption-based image retrieval -- by making only minor additions to the base architecture. We observe significant improvements across tasks compared to existing task-specific models -- achieving state-of-the-art on all four tasks. Our work represents a shift away from learning groundings between vision and language only as part of task training and towards treating visual grounding as a pretrainable and transferable capability.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Quoc V. Le
- Yoshua Bengio
- Yejin Choi
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Furu Wei
- Mohit Bansal
- Noah A. Smith
- Hinrich Schütze
- Jason Aaron Edward Weston
In Multimodal Learning
Citations in the corpus
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or unified text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, neutral): “Finally, CLIP is related to a recent burst of activity on learning joint models of vision and language (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2019; Li et al., 2020b; Yu et al., 2020).”
- Experience Grounds Language (2020, background): “These combined text and vision benchmarks are rich enough to train large-scale, multimodal transformers (Li et al., 2019a; Lu et al., 2019; Zhou et al., 2019) without language pretraining, for example via conceptual captions (Sharma et al., 2018), or further broadened to include audio (Tsai et al.,…”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized representations for image regions and textual words through a single Transformer.”
- Evaluating Large Language Models Trained on Code (2021, background): “…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
- VisualBERT: A Simple and Performant Baseline for Vision and Language (2019, methodology): “Concurrently with our work, ViLBERT (Jiasen et al., 2019) proposes to learn joint representation of images and text using a BERT-like architecture but has separate Transformers for vision and language that can only attend to each-other (resulting in twice the parameters).”
- A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “LXMERT has been shown to outperform ViLBERT and VisualBERT on downstream tasks such as visual reasoning tasks.”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, background): “Another line of work related to ours is visual-linguistic representation learning (Lu et al., 2019; Tan and Bansal, 2019; Su et al., 2020).”
- Learning Visual Representations with Caption Annotations (2020, methodology): “Here, we contextualize token representations among visual ones by fusing them in a data-driven manner (similar to [40]).”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Motivated by pre-trained models like BERT [1] and ERNIE [2] which have significantly improved the performance on many NLP tasks, researchers ( [3] [4] [5] [6] [7] [8] [9] [10]) have noticed the importance of pre-training for vision-language tasks, e.g., Visual Question Answering(VQA) [11] and Visual…”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “of BERT [13] on natural language understanding, there has been a surging research interest in developing multimodal pre-training methods for vision-and-language representation learning (e.g., ViLBERT [38], LXMERT [65], and UNITER [12]). When finetuned on downstream tasks, these pre-trained models hav”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, neutral): “These can be compared to more sophisticated multimodal methods, such as supervised multimodal bitransformers [40] using either Image-Grid or Image-Region features ( MMBT-Grid and MMBT-Region ), and versions of ViLBERT [50] and Visual BERT [46] that were only unimodally pretrained and not pretrained ”
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “e is an growing interest in pre-training generic models to solve a variety of V+L problems, such as visual questionanswering (VQA), image-text retrieval and image captioning etc. The existing methods [38,39,23,5,47,36,19,10] employ BERT-like objectives [6] to learn crossmodal representations from a ”
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “CO[5] and Flickr30k[6] datasets. 7 Method Flickr30k MSCOCO Image Retrieval Sentence Retrieval Image Retrieval Sentence Retrieval R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 1k Test set ViLBERT[14] 31.9 61.1 72.8 - - - - - - - - - Unicoder-VL[18] 48.4 76.0 85.2 64.3 85.8 92.3 43.4 76.0 87.0 5”
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, background): “Since our EMNLP submission, a few other useful preprints have recently been released (in August) on similar cross-modality pre-training directions: ViLBERT (Lu et al., 2019) and VisualBERT (Li et al.”
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “The same experiment results can be observed in ViLBERT (Lu et al. 2019) and UNITER (Chen et al. 2019).”
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “Inspired by the BERT model [6], a number of methods attempt to learn generic multi-modal fusion layers, given the pre-extracted visual region features and textual encoder [21, 24, 26, 31, 43, 57].”