Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization

By Hantao Yao, Rui Zhang, Changsheng Xu. Computer Vision and Pattern Recognition, 2023

466 citations recorded by the source corpus.

Prompt tuning is an effective way to adapt the pretrained visual-language model (VLM) to the downstream task using task-related textual tokens. Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. However, the specific textual knowledge is worse generalization to the unseen classes because it forgets the essential general textual knowledge having a strong generalization ability. To tackle this issue, we introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes. The key insight of KgCoOp is that the forgetting about essential knowledge can be alleviated by reducing the discrepancy between the learnable prompt and the hand-crafted prompt. Especially, KgCoOp minimizes the discrepancy between the textual embeddings generated by learned prompts and the hand-crafted prompts. Finally, adding the KgCoOp upon the contrastive loss can make a discriminative prompt for both seen and unseen tasks. Extensive evaluation of several benchmarks demonstrates that the proposed Knowledge-guided Context Optimization is an efficient method for prompt tuning, i.e., achieves better performance with less training time. code.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/88779e873b7ec860d6b6a4c2ddfc28dd67c86b67

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sheng Jin
2. Shijian Lu
3. Jingyi Zhang
4. Jiaxing Huang

In Computer Vision and Pattern Recognition

1. Sheng Jin
2. Shijian Lu
3. Jingyi Zhang
4. Jiaxing Huang

Citations in the corpus

• Vision-Language Models for Vision Tasks: A Survey (2023, background): “KgCoOp [145] [code] TPT Few-shot Sup.”
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, unknown): “Prompt tuning methods [2, 55, 63] can efficiently adapt VFMs towards dedicated scenarios by designing suitable prompts.”