SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

By Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, Yuan Cao. 2021

965 citations recorded by the source corpus.

With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Multimodal Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “, CLIP [71]) have been trained on joint image(s) and corresponding text captions via variants of a masked language modeling objective [72, 73, 74, 75], a contrastive loss [76, 77, 71] or other supervised objectives[78, 79].”
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (2022, background): “For example, pre-training with multi-modal data [27, 52], web hypertext [2], or web instruction-action mapping [38] could help agents better understand and leverage rich semantics of webpage content, actions, and instructions.”
Survey of Hallucination in Natural Language Generation (2022, background): “With the vast advancement of the Transformer architecture [35, 189] in both CV and NLP, there is a trend to pre-train large-scale unified vision-language (VL) models [1, 27, 105, 196, 197, 200] to perform vision grounded text generation tasks, such as image captioning and visual question answering (”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “For example, in terms of linguistic data, visual-language models (VLMs) (Wang et al. 2021; Jain et al. 2021) are trained on image and video captions, but large language models (LMs) (Devlin et al. 2018; Thoppilan et al. 2022; Chen et al. 2021) are additionally trained on a large corpora of other…”
Flamingo: a Visual Language Model for Few-Shot Learning (2022, background): “1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, methodology): “, 2020), † are from (Iki and Aizawa, 2021), and ‡ are from (Wang et al., 2021).”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “, 2021) nor encoder-decoder models (Cho et al., 2021; Wang et al., 2021) can excel at both types of tasks, whereas a single unified encoder-decoder (Zhou et al.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “2Recent papers propose exceptions, like generating masked-out spans [21] or text [77, 116], but it is unclear whether they can outperform the ‘VisualBert’s on vision-language tasks like VCR [124].”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, methodology): “SimVLM [55], while showing impressive performance, requires a prohibitively large amount of data for pretraining and the language and vision components to be trained in tandem.”
FLAVA: A Foundational Language And Vision Alignment Model (2021, methodology): “FLAVA also has comparable performance to SimVLM [109] (Table 5 row 3) on language tasks while underper- forming it on multimodal tasks and ImageNet linear evaluation.”
Scaling Up Vision-Language Pretraining for Image Captioning (2021, methodology): “Z: the pre-trained model without finetuning, where we add the prefix “a picture of” during inference as the prompt to improve the quality of zero-shot generation following [43].”
Florence: A New Foundation Model for Computer Vision (2021, methodology): “Recently, there is an increasing trend (Huang et al., 2021; Xue et al., 2021; Wang et al., 2021; Kim et al., 2021; Dou et al., 2021) of end-to-end approaches to reduce dependency on the object bounding box, which instead consider grid-based feature representations as the fine-grained features for…”
ClipCap: CLIP Prefix for Image Captioning (2021, background): “[40] mitigate the need for supplementary annotations, but still perform an extensive pre-train process with millions of imagetext pairs, resulting in a lengthy training time.”
Achieving Human Parity on Visual Question Answering (2021, methodology): “Adding the 6The size of pre-training data we use is more than 100 times smaller than SimVLM.”
FILIP: Fine-grained Interactive Language-Image Pre-Training (2021, methodology): “We compare our FILIP model against methods with complex attention layers including Unicoder-VL (Li et al., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020), ERNIE-ViL (Yu et al., 2021), Oscar (Li et al., 2020b), VinVL (Zhang et al., 2021), ALBEF (Li et al., 2”
VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts (2021, methodology): “The second category models the interaction of images and text using a deep fusion encoder with cross-modal attention [43, 30, 41, 24, 51, 3, 26, 25, 14, 49, 16, 17, 20, 23, 46].”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, background): “A line of works [36, 46, 50] train both visual and textual encoders from scratch, leading to a high computation cost.”
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models (2023, neutral): “…et al., 2021; Jia et al., 2021), the fusion-encoder architecture (Tan & Bansal, 2019; Li et al., 2021), the encoder-decoder architecture (Cho et al., 2021; Wang et al., 2021b; Chen et al., 2022b), and more recently, the unified transformer architecture (Li et al., 2022; Wang et al., 2022b).”
Generalized Decoding for Pixel, Image, and Language (2022, background): “Recently, researchers [63,73,74] have found that image-text data at scale can be helpful for visual representation learning ( e.g ., enabling zero-shot image classification [32, 59], action recognition [85,88], and image generation [44]).”
InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, methodology): “Techniques for combining pretrained models include joint finetuning on a new task [4, 33, 40, 67], communication through prompting [62, 69], composing probability distributions of energybased models [11, 37], guiding one model with feedback from another [61], and iterative optimization [34].”