LXMERT: Learning Cross-Modality Encoder Representations from Transformers

By Hao Hao Tan, Mohit Bansal. 2019

2,971 citations recorded by the source corpus.

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Yejin Choi
  4. Percy Liang
  5. Luke S. Zettlemoyer
  6. Zhe Gan
  7. Caiming Xiong
  8. Yoshua Bengio

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Cordelia Schmid
  7. Jitendra Malik
  8. Andrew Zisserman

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Linjie Li
  4. Jianfeng Gao
  5. Mohit Bansal
  6. Zicheng Liu
  7. Devi Parikh
  8. Jianfeng Wang

In Representation & Self-Supervised Learning

  1. Andrew Zisserman
  2. Andrea Vedaldi
  3. Ishan Misra
  4. Trevor Darrell
  5. Jianfeng Gao
  6. Cordelia Schmid
  7. Mohit Bansal
  8. Armand Joulin

Citations in the corpus