Conditional Prompt Learning for Vision-Language Models
By Kaiyang Zhou, Jingkang Yang, Chen Change Loy, Ziwei Liu. 2022
2,313 citations recorded by the source corpus.
With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning—a recent trend in NLP—to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even showing promising transferability beyond a single dataset; and yields stronger domain generalization performance as well. Code is available at https://github.com/KaiyangZhou/CoOp.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ziwei Liu
- Chen Change Loy
- Yu Qiao
- Hongsheng Li
- Peng Gao
- Tao A. Xiang
- Fahad Shahbaz Khan
- Ming-Hsuan Yang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Ming-Hsuan Yang
- Xiaogang Wang
- Dacheng Tao
- Chen Change Loy
- Yi Yang
- Yu Qiao
- Trevor Darrell
In Prompting & Reasoning
In Multimodal Learning
Citations in the corpus
- Learning to Prompt for Vision-Language Models (2021, methodology): “…pave the way for future research on efficient adaptation methods for emerging foundation models, which is still a nascent research topic. benchmark (Zhou et al., 2022b) contains 7 image recognition datasets, which cover a wide range of classification problems, such as generic object recognition,…”
- OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “Our motivation behind using Qctx is to learn a unified textual context [62, 63] for a sample image.”
- S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “There have been emerging a bunch of prompting methods, such as [35, 38, 26, 76, 77, 3, 24, 66, 14], to name a few 6 .”
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “CoOp [93] and CoCoOp [94] apply prefix tuning for adapting the CLIP model to various image recognition tasks.”
- Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, methodology): “Subsequently, to inject textual branch with visual signals, CoCoOp [83] and VT-CLIP [78] propose to train a intermediate network to generate image tokens as conditional inputs for the textual vectors.”
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, background): “The success of efficient tuning strategies in NLP promotes their applications on vision-language models [22, 34, 65, 76, 79] like CLIP [22].”
- Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Users primarily engage with generative AI models through natural language instructions, known as text prompts [73, 74].”
- Generalized Out-of-Distribution Detection: A Survey (2021, neutral): “Further, full-spectrum OOD detection (Yang et al., 2022b) highlights the effects of “covariate-shifted in-distribution”, and show that most of the previous OOD detectors are unfortunately sensitive to covariate shift rather than semantic shift.”
- Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, methodology): “We compare the performance of our approach with zero-shot CLIP [35], CoOp [59], CoCoOp [58], ProDA [28] and MaPLe [22], in Table 1.”
- Bridging CLIP and StyleGAN through Latent Alignment for Image Editing (2022, neutral): “In recent years, vision-language pre-training [14, 26, 29, 36, 47, 48] has become an attractive novel research topic.”
- RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (2022, background): “2021) or explored novel applications (e.g., Tan et al., 2022; Zhou et al., 2022; Levine et al., 2022).”
- Transformer-Based Visual Segmentation: A Survey (2023, background): “To further handle inflexible one-hot common taxonomy, LMSeg [300] extends such embedding with learnable tokens [265] and proposes a dataset-specific augmentation for each dataset.”