CLIP Models are Few-Shot Learners: Empirical Studies on VQA and Visual Entailment

By Haoyu Song, Li Dong, Weinan Zhang, Ting-Wei Liu, Furu Wei. 2022

174 citations recorded by the source corpus.

CLIP has shown a remarkable zero-shot capability on a wide range of vision tasks. Previously, CLIP is only regarded as a powerful visual encoder. However, after being pre-trained by language supervision from a large amount of image-caption pairs, CLIP itself should also have acquired some few-shot abilities for vision-language tasks. In this work, we empirically show that CLIP can be a strong vision-language few-shot learner by leveraging the power of language. We first evaluate CLIP’s zero-shot performance on a typical visual question answering task and demonstrate a zero-shot cross-modality transfer capability of CLIP on the visual entailment task. Then we propose a parameter-efficient fine-tuning strategy to boost the few-shot performance on the vqa task. We achieve competitive zero/few-shot results on the visual question answering and visual entailment tasks without introducing any additional pre-training procedure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Andy Zeng
3. Brian Ichter
4. Peter R. Florence
5. Li Fei-Fei
6. Chelsea Finn
7. Michael S. Ryoo
8. Dhruv Shah

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Li Fei-Fei
3. Jitendra Malik
4. Sergey Levine
5. Dacheng Tao
6. Kristen Grauman
7. Andy Zeng
8. Trevor Darrell

In Multimodal Learning

1. Sergey Levine
2. Michael S. Ryoo
3. Brian Ichter
4. Andy Zeng

In Representation & Self-Supervised Learning

1. Andy Zeng
2. Johnny Lee
3. Peter R. Florence
4. Krzysztof Marcin Choromanski
5. Vikas Sindhwani
6. Sergey Levine
7. Michael S. Ryoo
8. Dhruv Shah

Citations in the corpus

• Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “, (Song et al. 2022a) and ALMs and LMs for speech and text modeling e.”
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action (2022, background): “…1M demonstrations — annotated with fine-grained language labels in indoor [1, 3–5, 22 ] Recent progress in using large-scale models of natural language and images trained on diverse data has enabled applications in a wide variety of textual [31–33], visual [13, 34–38], and embodied domains [39–44].”
• RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Beyond classification, CLIP models also exhibit the flexibility to be integrated into a variety of vision-language applications, including detection [33], [34], segmentation [35], [36], captioning [37], [38], VQA [39], [40], and conditional image generation [41].”
• VLP: A Survey on Vision-language Pre-training (2022, unknown): “Visual Entailment (VE) [66, 67, 68] .”
• LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, unknown): “…reasoning ability of VLMs, including VQA [22, 28] requiring models to answer from visual contents and textual questions, and Visual Entailment [51] requiring models to determine the consistency of text descriptions and visual contents, etc. Traditional vision-language models employ neural…”