Learning to Prompt for Vision-Language Models
By Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu. 2021
4,174 citations recorded by the source corpus.
Large pre-trained vision-language models like CLIP have shown great potential in learning representations that are transferable across a wide range of downstream tasks. Different from the traditional representation learning that is based mostly on discretized labels, vision-language pre-training aligns images and texts in a common feature space, which allows zero-shot transfer to a downstream task via prompting, i.e., classification weights are synthesized from natural language describing classes of interest. In this work, we show that a major challenge for deploying such models in practice is prompt engineering, which requires domain expertise and is extremely time-consuming—one needs to spend a significant amount of time on words tuning since a slight change in wording could have a huge impact on performance. Inspired by recent advances in prompt learning research in natural language processing (NLP), we propose Context Optimization (CoOp), a simple approach specifically for adapting CLIP-like vision-language models for downstream image recognition. Concretely, CoOp models a prompt’s context words with learnable vectors while the entire pre-trained parameters are kept fixed. To handle different image recognition tasks, we provide two implementations of CoOp: unified context and class-specific context. Through extensive experiments on 11 datasets, we demonstrate that CoOp requires as few as one or two shots to beat hand-crafted prompts with a decent margin and is able to gain significant improvements over prompt engineering with more shots, e.g., with 16 shots the average gain is around 15% (with the highest reaching over 45%). Despite being a learning-based approach, CoOp achieves superb domain generalization performance compared with the zero-shot model using hand-crafted prompts.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Quoc V. Le
- Yejin Choi
- Yoshua Bengio
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Yi Yang
- Andrew Zisserman
In Prompting & Reasoning
- Hannaneh Hajishirzi
- Tushar Khot
- Ashish Sabharwal
- Hongsheng Li
- Ziwei Liu
- Yejin Choi
- Mohit Bansal
- Denny Zhou
Citations in the corpus
- CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2021, background): “, 2021), of zero-shot prompts (Zhou et al., 2021) and of adversarial attacks to uncurated datasets (Carlini & Terzis, 2021).”
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Recent work has explored alternative strategies for adapting these models to specific target tasks (Zhou et al., 2021; Gao et al., 2021; Zhang et al., 2021), for instance via a lightweight residual feature adapter.”
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, neutral): “Some concurrent works also extend parameter-efficient techniques to CLIP models [2,54,55].”
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “Inspired by the success of prompt tuning in NLP, [77] applies prompt tuning to visual-linguistic pre-trained models ( e.g ., CLIP [55]) to perform few-shot image classification.”
- Grounded Language-Image Pre-training (2021, background): “Different from CLIP that fuses vision and language only at the last dot product layer [45], we show that deep cross-modality fusion applied by GLIP, as shown in Figure 2 (Middle), is crucial to learn high-quality language-aware visual representations and to achieve superior transfer learning perform”
- OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “Our motivation behind using Qctx is to learn a unified textual context [48, 49] for a sample image.”
- S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “DyTox [14] and L2P [76] are two main baselines in this paper, and we list the parameters below for reference.”
- CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “Although CoOp only uses a small number of parameters, its learnable prompts are equipped in front of the text encoder and require both the forward and backward propagation, as shown by green dotted lines in Figure 1.”
- Conditional Prompt Learning for Vision-Language Models (2022, methodology): “[62], recent successes in visionlanguage models are mainly attributed to the developments in i) Transformers [48], ii) contrastive representation learning [4, 17, 20], and iii) web-scale training datasets [24, 40].”
- Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, methodology): “Furthermore, CoOp [70], CLIP-Adapter [17] and WiSE-FT [64] indicate that the performance of CLIP can be significantly improved with either abundant or limited training data by fine-tuning weight-”
- Robust fine-tuning of zero-shot models (2021, methodology): “3, including distillation, additional regularization, and CoOp [110].”
- Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, methodology): “Surprisingly, with this well-designed cache model, Tip-Adapter without fine-tuning can achieve comparable performance compared to those training-required approaches, including CoOp [73] and CLIP-Adapter [16].”
- Incremental Prototype Prompt-tuning with Pre-trained Representation for Class Incremental Learning (2022, background): “Context optimization (CoOp) [44] models context words by using continuous vectors learned from data end-to-end, avoiding manual prompt adjustment.”
- Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification (2022, background): “Significant efforts have been devoted to prompting vision language models [12, 14, 29, 43, 46, 73, 74].”
- AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (2023, positive): “The SOTA competing methods include CLIP (Radford et al., 2021), CLIP-AC (Radford et al., 2021), Win-CLIP (Jeong et al., 2023), VAND (Chen et al., 2023), and CoOp (Zhou et al., 2022b).”
- RemoteCLIP: A Vision Language Foundation Model for Remote Sensing (2023, background): “Recent advancements in CLIP models have primarily concentrated on scaling the model size and data size [53], incorporating self-supervision [54], [55], [56], enhancing pretraining efficiency [57], [58], and few-shot adaptation [59], [60], among others.”
- Image Segmentation Using Text and Image Prompts (2021, background): “Zhou et al. [45] propose context optimization (CoOp) which automatically learns tokens that perform well for given downstream tasks.”