LXMERT: Learning Cross-Modality Encoder Representations from Transformers
By Hao Hao Tan, Mohit Bansal. 2019
2,971 citations recorded by the source corpus.
Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Percy Liang
- Luke S. Zettlemoyer
- Zhe Gan
- Caiming Xiong
- Yoshua Bengio
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Cordelia Schmid
- Jitendra Malik
- Andrew Zisserman
In Multimodal Learning
In Representation & Self-Supervised Learning
- Andrew Zisserman
- Andrea Vedaldi
- Ishan Misra
- Trevor Darrell
- Jianfeng Gao
- Cordelia Schmid
- Mohit Bansal
- Armand Joulin
Citations in the corpus
- Learning Transferable Visual Models From Natural Language Supervision (2021, neutral): “Finally, CLIP is related to a recent burst of activity on learning joint models of vision and language (Lu et al., 2019; Tan & Bansal, 2019; Chen et al., 2019; Li et al., 2020b; Yu et al., 2020).”
- UNITER: Learning UNiversal Image-TExt Representations (2019, methodology): “Compared with LXMERT (Tan & Bansal, 2019) and ViLBERT (Lu et al., 2019) that use two streams (one Transformer for each modality), our UNITER model can learn joint contextualized representations for image regions and textual words through a single Transformer.”
- A Multimodal Framework for the Detection of Hateful Memes (2020, methodology): “Early-fusion architectures were shown to outperform the late-fusion ones for the task [15]; therefore, we focus on three early-fusion pretrained models, namely LXMERT [34], UNITER [3] and Oscar [20].”
- Contrastive Learning of Medical Visual Representations from Paired Images and Text (2020, methodology): “• Contrastive-Binary-Loss : This baseline differs from ConVIRT by contrasting the paired image and text representations with a binary classification head, as is widely done in visual-linguistic pretraining work (Tan and Bansal, 2019; Su et al., 2020).”
- Learning Visual Representations with Caption Annotations (2020, background): “Building on the recent advances in natural language processing [14,59], several works have fine-tuned BERT [14] to fuse visual and textual information [40,55,56,57,70] for VL tasks.”
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “2019) and LXMERT (Tan and Bansal 2019) were based on two-stream cross-modal Transformers, which brings more specific representations for images and texts.”
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, methodology): “And to demonstrate the versatility of VILLA, we further apply it to another V+L model LXMERT [65] with a different architecture design from UNITER (two-stream vs.”
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks (2020, methodology): “e is an growing interest in pre-training generic models to solve a variety of V+L problems, such as visual questionanswering (VQA), image-text retrieval and image captioning etc. The existing methods [38,39,23,5,47,36,19,10] employ BERT-like objectives [6] to learn crossmodal representations from a ”
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data (2020, methodology): “LXMERT[15] added some VQA training data into pre-training and obtained state-of-the-art results on VQA task.”
- CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP (2022, background): “Vision-Language Modelling Pre-training methods that have recently moved from raw text to multi-modal data (e.g., image-text) have revolutionized numerous multimodal tasks (e.g., image-text matching [9, 27, 34], image captioning [44], Visual Question Answering [9,20,33,34]).”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “Recently, V&L pre-training has been proposed as an effective technique to improve the performance on various V&L tasks (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019a; Su et al., 2019; Chen et al., 2020; Zhou et al., 2020; Huang et al., 2020; Li et al., 2020; Zhang et al., 2021; Li et al.,…”
- StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery (2021, background): “Joint representations Multiple works learn cross-modal Vision and language (VL) representations [8, 39, 44, 29, 24, 43, 23, 4, 26] for a variety of tasks, such as languagebased image retrieval, image captioning, and visual question answering.”
- Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
- MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Our approach combines past success at matching images to their captions [91, 62] along with ‘VisualBERT’-style prediction of independent tokens [106, 20] – though, crucially,”
- KAT: A Knowledge Augmented Transformer for Vision-and-Language (2021, background): “, 2019) and LXMERT (Tan and Bansal, 2019) propose a two-stream architecture to process images and text independently and fused by a third transformer in ta later stage.”