ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
By Wonjae Kim, Bokyung Son, Ildoo Kim. 2021
2,347 citations recorded by the source corpus.
Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Cordelia Schmid
- Jitendra Malik
- Yi Yang
In Multimodal Learning
In Model Architectures
Citations in the corpus
- Towards Language-Free Training for Text-to-Image Generation (2021, methodology): “Inspired by the BERT model [6], a number of methods attempt to learn generic multi-modal fusion layers, given the pre-extracted visual region features and textual encoder [21, 24, 26, 31, 43, 57].”
- How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “V&L models designed for these tasks often consist of a visual encoder, a text encoder, and a cross-modal interaction module (Kim et al., 2021).”
- All in One: Exploring Unified Video-Language Pre-Training (2022, methodology): “Following the common practices [8, 17], we randomly mask text tokens with the probability of 0.”
- Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “Single-stream107 models (Chen et al., 2020; Li et al., 2019; Ramesh 108 et al., 2021; Lin et al., 2021; Kim et al., 2021a) 109 concatenate the patch-wise or regional visual fea- 110 tures and textual embeddings and feed them into a 111 single model.”
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “, 2020), using a ViT is more computation-friendly and has been adopted by the more recent methods (Li et al., 2021a; Kim et al., 2021).”
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “[24] attempts to eliminate the need for object detectors by directly extracting features from the raw pixels with CNNs. [1, 22, 28, 56] take a further step by applying Transform-ers to raw image patches and word tokens.”
- Scaling Up Vision-Language Pretraining for Image Captioning (2021, background): “More recently, instead of using object detectors for image feature extraction, end-to-end VLP based on convolution networks and transformers are becoming popular [14, 15, 19, 23, 44].”
- Florence: A New Foundation Model for Computer Vision (2021, background): “Recently, there is an increasing trend (Huang et al., 2021; Xue et al., 2021; Wang et al., 2021; Kim et al., 2021; Dou et al., 2021) of end-to-end approaches to reduce dependency on the object bounding box, which instead consider grid-based feature representations as the fine-grained features for…”
- UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “Our UFO-B/32 shares the same backbone with ViLT [25], but achieves significantly stronger performance except on NLVR(2), where our approach is slightly better.”
- FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “, 2021a) requires to be performed in an encoder-decoder structure, while the complexity of the self-attention (Kim et al., 2021) grows quadratically with the length of the prolonged concatenated sequences of both modalities.”
- VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “Following previous work [3, 20], our pre-training data consists of four image captioning datasets: Conceptual Captions (CC) [39], SBU Captions [32], COCO [27] and Visual Genome (VG) [21] datasets.”
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “…Lu et al., 2019; Xu et al., 2021), masked region classification/feature regression (Tan & Bansal, 2019; Chen et al., 2020b), object attribute prediction (Xu et al., 2021), contrastive loss (Li et al., 2020; 2021), word-region alignment (Chen et al., 2020b) word-patch alignment (Kim et al., 2021).”
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, methodology): “They achieve remarkable performance on image-text retrieval tasks, but lack the ability to model more complex interactions between image and text for other V+L tasks [21].”
- MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Some exceptions are Zhang et al. [125], who use an even larger object detector trained on more labeled data; Kim et al. [57], who use an ImageNet-pretrained backbone [26], and Shen et al. [100], who study a CLIP backbone [89] pretrained on web image-caption pairs.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with popular VQA methods [2, 9, 17, 18, 22, 23, 28, 53] and language models [5, 16, 59].”
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “Specifically, among the 8 question classes, Mutimodal-CoT Large achieves a 21.37% (67.43% → 88.80”
- Generalized Decoding for Pixel, Image, and Language (2022, background): “[36] Wonjae Kim, Bokyung Son, and Ildoo Kim.”