ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

By Wonjae Kim, Bokyung Son, Ildoo Kim. 2021

2,347 citations recorded by the source corpus.

Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Mohit Bansal
  3. Zhe Gan
  4. Yejin Choi
  5. Luke S. Zettlemoyer
  6. Percy Liang
  7. Furu Wei
  8. Caiming Xiong

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Cordelia Schmid
  7. Jitendra Malik
  8. Yi Yang

In Multimodal Learning

  1. Zhe Gan
  2. Lijuan Wang
  3. Jianfeng Gao
  4. Linjie Li
  5. Zicheng Liu
  6. Jianfeng Wang
  7. Mohit Bansal
  8. Xiaowei Hu

In Model Architectures

  1. Jianfeng Gao
  2. Jianwei Yang
  3. Lu Yuan
  4. Chunyuan Li
  5. Bin Xiao
  6. Xiyang Dai
  7. Pengchuan Zhang
  8. Yu Qiao

Citations in the corpus