CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

By Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-seng Chua, Maosong Sun. 2021

257 citations recorded by the source corpus.

Pre-Trained Vision-Language Models (VL-PTMs) have shown promising capabilities in grounding natural language in image data, facilitating a broad variety of cross-modal tasks. However, we note that there exists a significant gap between the objective forms of model pre-training and fine-tuning, resulting in a need for large amounts of labeled data to stimulate the visual grounding capability of VL-PTMs for downstream tasks. To address the challenge, we present Cross-modal Prompt Tuning (CPT, alternatively, Colorful Prompt Tuning), a novel paradigm for tuning VL-PTMs, which reformulates visual grounding into a fill-in-the-blank problem with color-based co-referential markers in image and text, maximally mitigating the gap. In this way, CPT enables strong few-shot and even zero-shot visual grounding capabilities of VL-PTMs. Comprehensive experimental results show that the prompt-tuned VL-PTMs outperform their fine-tuned counterparts by a large margin (e.g., 17.3% absolute accuracy improvement, and 73.8% relative standard deviation reduction on average with one shot in RefCOCO evaluation). We make the data and code for this paper publicly available at https://github.com/thunlp/CPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models (2022, background): “It has inspired a few recent works for language-augmented visual models [75, 59, 70, 18].”
Visual Prompt Tuning (2022, background): “Although prompting has also been applied to visionlanguage models recently [61,85,35,77,18], prompting is still limited to the input of text encoders.”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “As the earliest attempt, one type of them focus on tuning pretrained language or vision-language models with natural language discrete prompts [1, 17, 61, 65, 68, 69] or prompt engineering by mining or generating proper natural language discrete prompts [17, 29, 53].”
Conditional Prompt Learning for Vision-Language Models (2022, background): “With the rise of such powerful vision-language models, the community has recently started to investigate potential solutions to efﬁciently adapt these models to downstream datasets [14, 53, 56, 63].”
The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning (2022, methodology): “We modify CLIP so that it can incorporate the bounding box as input: inspired by a similar process from [70, 76], to pass a region to CLIP, we simply draw a bounding box on an image in pixel space—we use a green-bordered / opaque purple box as depicted in Figure 4b (this proved more effective than m”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “Many works have demonstrated its effectiveness [76, 135, 261, 264, 265] in CV and NLP tasks.”
Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models (2022, methodology): “CPT [15] converts visual grounding into a ll-in-the-blank problem by creating visual prompts with colored blocks and color-based textual prompts.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “CPT [217] is a color prompt tuning and uses different color modules to distinguish between entities by constructing the corresponding prompt templates.”
Prompt-aligned Gradient for Prompt Tuning (2022, background): “Thanks to the powerful language modeling of VLMs, we can establish a communication channel in human-readable natural language, i.e ., prompt [25, 51, 19], to query the general knowledge.”
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (2022, background): “These models exhibit remarkable results and transferring capabilities over many visual recognition tasks [44, 46].”
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization (2023, background): “ing [4] [10] [19] [22] [28] [30] [33] [38] has been proposed to adapt the pretrained VLM to downstream tasks, achieving a fantastic performance on various few-shot or zero-shot visual recognization tasks.”
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting (2021, background): “The work most related to ours is CPT [51], which reformulates dense predictions into a fill-in-the-blank problem by jointly marking co-referential parts of both image and text in color.”
MaPLe: Multi-modal Prompt Learning (2022, background): “Designing high quality contextual prompts have been proven to enhance the performance of CLIP and other V-L models [17, 42].”
Incremental Prototype Prompt-tuning with Pre-trained Representation for Class Incremental Learning (2022, background): “CPT [41] redefines visual grounding as the problem of filling in the blank of color-based co-referential markers in images and texts, mitigating the gap between the objective forms of model pre-training and fine-tuning.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “It has achieved satisfied performance in both NLP tasks [24, 27, 35] and CV tasks [47, 50, 57].”
Exploring Visual Prompts for Adapting Large-Scale Models (2022, unknown): “CPT [46] converts visual grounding into a ﬁll-in-the-blank problem by creating visual prompts with colored blocks and color-based textual prompts.”
Segment Anything in High Quality (2023, methodology): “For vision-based foundation models [21, 42, 56], prompt engineering [54, 44, 48, 55] that freezes the pre-trained model is first explored in vision-language models, such as CLIP [35].”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “…to overcome the limitations imposed by pre-defined tasks, resulting in a gap in open-ended task capabilities compared to LLMs. Recently, visual prompt tuning [24, 66, 70, 67, 54] has emerged as a way to flexibly outline some pure vision tasks (see Figure 1b), such as object detection, instance”