VisualBERT: A Simple and Performant Baseline for Vision and Language
By Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang. 2019
2,355 citations recorded by the source corpus.
We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Yoshua Bengio
- Zhe Gan
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Multimodal Learning
Citations in the corpus
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “…for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or unified text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et al., 2019).”
- Experience Grounds Language (2020, methodology): “These combined text and vision benchmarks are rich enough to train large-scale, multimodal transformers (Li et al., 2019a; Lu et al., 2019; Zhou et al., 2019) without language pretraining (e.”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Concurrent pre-trained models include: ViLBERT, LXMERT, Unicoder-VL, VisualBERT and VLBERT.”
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “VisualBERT [46] directly extend BERT [12] for vision and language domain.”
- A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “Early-fusion methods such as MMBT [14], VisualBERT [19], and ViLBERT [23], on the other hand, employ more complex approaches to process the two modalities jointly within the model architecture.”
- TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “For the former, we refine the pre-training tasks in VLP [37, 34] to support the extra scene text input.”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, background): “…models like BERT [1] and ERNIE [2] which have significantly improved the performance on many NLP tasks, researchers ( [3] [4] [5] [6] [7] [8] [9] [10]) have noticed the importance of pre-training for vision-language tasks, e.g., Visual Question Answering(VQA) [11] and Visual Commonsense…”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “Compared to this two-stream architecture, recent work such as VL-BERT [60], VisualBERT [33], B2T2 [1], UnicoderVL [30] and UNITER [12] advocate a single-stream model design, where two modalities are directly fused in early stage.”
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (2020, positive): “We include results for simple fusion methods where we take the mean of the unimodal ResNet-152 and BERT output scores ( Late Fusion ) or concatenate ResNet-152 features with BERT and training an MLP on top ( Concat BERT ).”
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, background): “Recent studies [23,39,5,36,20,19,47] on vision-language pretraining (VLP) have shown it effective to learn generic representations from massive image-text pairs, and that fine-tuning VLP models on task-specific data achieves state-of-the-art (SoTA) results on well-established V+L tasks.”
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “For example, most recent cross-modal pre-trained models [16, 17, 18, 19, 20, 21] use following 2 datasets in pre-training: The Conceptual Captions (CC) dataset[2], which contains 3M images with descriptions harvested from the Alt-text HTML attribute of the web pages, and SBU Captions[3], which consi”
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training (2019, methodology): “3) VisualBERT is pre-trained on MSCOCO Captions dataset.”
- Training data-efficient image transformers & distillation through attention (2020, background): “Nevertheless hybrid architectures that combine convnets and transformers, including the self-attention mechanism, have recently exhibited competitive results in image classification [53], detection [6, 26], video processing [42, 50], unsupervised object discovery [33], and unified text-vision tasks ”
- CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “Overall, we evaluated six bias tests for gender and seven bias tests for race and found that our family of CM3 models show significantly less bias than other models, speicifically VisualBERT (Li et al., 2019) and ViLBert (Lu et al., 2019).”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “…V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b).”
- StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Recent works have studied how to ground modern language models, by training them to accept additional environment inputs [23, 24, 25, 26, 27, 28, 29] or to directly output actions [30, 31, 32].”