Prompt Distribution Learning

By Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, Xinmei Tian. Computer Vision and Pattern Recognition, 2022

353 citations recorded by the source corpus.

We present prompt distribution learning for effectively adapting a pre-trained vision-language model to address downstream recognition tasks. Our method not only learns low-bias prompts from a few samples but also captures the distribution of diverse prompts to handle the varying visual representations. In this way, we provide high-quality task-related content for facilitating recognition. This prompt distribution learning is realized by an efficient approach that learns the output embeddings of prompts instead of the input embeddings. Thus, we can employ a Gaussian distribution to model them effectively and derive a surrogate loss for efficient training. Extensive experiments on 12 datasets demonstrate that our method consistently and significantly outperforms existing methods. For example, with 1 sample per category, it relatively improves the average result by 9.1% compared to human-crafted prompts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e8df3224f6db7e839d6175d8d8324ab1d808e1c2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fahad Shahbaz Khan
2. Salman Hameed Khan
3. Muzammal Naseer
4. Ming-Hsuan Yang
5. Sheng Jin
6. Shijian Lu
7. Yuehua Wu
8. Hanoona Abdul Rasheed

In Computer Vision and Pattern Recognition

1. Fahad Shahbaz Khan
2. Salman Hameed Khan
3. Ming-Hsuan Yang
4. Luc Van Gool
5. Martin Danelljan
6. Muzammal Naseer
7. Philip H. S. Torr
8. Shijian Lu

Citations in the corpus

• Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, methodology): “We compare the performance of our approach with zero-shot CLIP [35], CoOp [59], CoCoOp [58], ProDA [28] and MaPLe [22], in Table 1.”
• Vision-Language Models for Vision Tasks: A Survey (2023, background): “ProDA [134] TPT Few-shot Sup.”
• Prompt-aligned Gradient for Prompt Tuning (2022, methodology): “ProDA [29] learns a prompt distribution over the output embedding space.”
• MaPLe: Multi-modal Prompt Learning (2022, background): “To address the above challenges, existing works propose language prompt learning to avoid manually adjusting the prompt templates and providing a mechanism to adapt the model while keeping the original weights frozen [14, 25, 29, 48, 49].”