Exploring Visual Prompts for Adapting Large-Scale Models

By Hyojin Bahng, Ali Jahanian, S. Sankaranarayanan, Phillip Isola. 2022

380 citations recorded by the source corpus.

We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance. The surprising effectiveness of visual prompting provides a new perspective on adapting pre-trained models in vision. Code is available at http://hjbahng.github.io/visual_prompting .

Subfields: Biophysics, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/aa8c61c9f6bb21c57e49611ccb995cfda1b53b10

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

1. Sheng Jin
2. Huchuan Lu
3. Carl Vondrick
4. Hang Zhang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Huchuan Lu
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Dong Wang
6. Shijian Lu
7. Dacheng Tao
8. Bichen Wu

In Electrical and Electronic Engineering

1. Huchuan Lu
2. Jiaxing Huang
3. Xin Chen
4. Shanghang Zhang
5. Bichen Wu
6. Dong Wang
7. Jaroslav Stejskal
8. Song Han

Citations in the corpus

• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, neutral): “Prompt tuning methods [2, 55, 63] can efficiently adapt VFMs towards dedicated scenarios by designing suitable prompts.”
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications (2024, neutral): “The notion of prompt engineering was initially investigated and popularized in the LLMs [Liu et al. , 2023], [Tonmoy et al. , 2024], [Chen et al. , 2023] later extended to VLMs [Wu et al. , 2023], [Bahng et al. , 2022].”
• Doubly Right Object Recognition: A Why Prompt for Visual Rationales (2022, background): “One advantage of the visual prompt is that it does not require model access at test time, which is flexible [4].”
• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “VP [147] [code] VPT Supervised Investigate the efficacy of visual prompt tuning for VLM transfer learning.”
• ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation (2023, background): “Furthermore, CTTA also encompasses the efficient zero-shot adaptation of foundation models to continual downstream tasks or distributions [2, 30].”
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, background): “Our mask prompt tuning is more relevant to prompt tuning in the visual domain (Bahng et al., 2022; Jia et al., 2022) where learnable vectors are applied to the image domain.”
• Visual Prompt Multi-Modal Tracking (2023, background): “Intuitively, there is a large inheritance between multi-modal and single RGB-modal tracking, which should share most of prior knowledge on feature extraction or attention patterns.”