How Much Can CLIP Benefit Vision-and-Language Tasks?

By Sheng Shen, Liunian Harold Li, Hao Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zi-Mei Yao, Kurt Keutzer. 2021

516 citations recorded by the source corpus.

Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation (2022, background): “Single-stream models (Chen et al., 2020; Li et al., 2019; Ramesh et al., 2021; Lin et al., 2021; Kim et al., 2021a; Shen et al., 2022) concatenate the patch-wise or regional visual features and textual embeddings and feed them into a single model.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, methodology): “We follow [45] to pre-train on COCO [29] and VG [21] images with multi-modal language modeling, Method VQA GQA NLVR visual question answering, and image-text matching.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “However, [44] show that the weights of contrastive language-image models contain useful semantic information for VL tasks.”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “While concurrent work (Shen et al., 2021) has explored building on top of models pretrained with such dataset, we focus on pretraining from scratch to explore the limit of generative VLP.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Some exceptions are Zhang et al. [125], who use an even larger object detector trained on more labeled data; Kim et al. [57], who use an ImageNet-pretrained backbone [26], and Shen et al. [100], who study a CLIP backbone [89] pretrained on web image-caption pairs.”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, background): “[95] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer.”
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm (2021, methodology): “Shen et al. (2021) also shows that the pretrained CLIP model can significantly benefit the downstream VQA and image caption tasks.”
Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain? (2021, methodology): “We investigate whether ﬁne-tuning CLIP for the MedVQA task beneﬁts medical VQA as much as it beneﬁts general-domain VQA, as observed in previous work Shen et al. [2021].”
A Cookbook of Self-Supervised Learning (2023, background): “While its advantages in vision-language downstream applications are evident [Shen et al., 2022, Dou et al., 2022], shared embedding spaces can also be constructed by training solely the vision encoder first, fixing it, and then training a matching language encoder, as described in [Zhai et al.,…”
Cross-lingual and Multilingual CLIP (2022, unknown): “…main contribution was image classification without predefined labels, CLIP representations have since proved useful for various other tasks such as Image-Text retrieval (Radford et al., 2021), Visual Question-Answering (Shen et al., 2021), and Automatic Image Captioning (Mokady et al., 2021).”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, methodology): “We evaluate results on eight downstream datasets covering a wide range of textual and video domains ( e.g. GIFs, YouTube videos, TV shows, movies), and multiple VideoQA paradigms: open-ended VideoQA ( iVQA [113], MSRVTT-QA [109], MSVD-QA [109], ActivityNet-QA [120] and TGIF-QA FrameQA [35]), multipl”
ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension (2022, methodology): “Shen et al. (2021) show that a simple zero-shot application of CLIP to visual question answering performs almost on par with random chance.”
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training (2022, methodology): “, 2019)-style VLP models (Lu et al., 2019; Tan and Bansal, 2019; Li et al., 2019; Chen et al., 2020; Shen et al., 2021) are trained to perform multimodal understanding tasks, using objectives like image-text matching and masked language modeling.”