Visual Prompt Tuning

By Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, Ser-Nam Lim. 2022

2,722 citations recorded by the source corpus.

The current modus operandi in adapting pre-trained models involves updating all the backbone parameters, ie, full fine-tuning. This paper introduces Visual Prompt Tuning (VPT) as an efficient and effective alternative to full fine-tuning for large-scale Transformer models in vision. Taking inspiration from recent advances in efficiently tuning large language models, VPT introduces only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the model backbone frozen. Via extensive experiments on a wide variety of downstream recognition tasks, we show that VPT achieves significant performance gains compared to other parameter efficient tuning protocols. Most importantly, VPT even outperforms full fine-tuning in many cases across model capacities and training data scales, while reducing per-task storage cost.

Subfields: Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Visual Instruction Tuning (2023, background): “To clarify, visual instruction tuning is different from visual prompt tuning [20]: the former aims to improve the model’s instruction-following abilities, while the latter aims to improve the parameter-efficiency in model adaptation.”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, neutral): “vision [22, 69] and vision-and-language models [26]).”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “As shown, our zero-initialized attention outperforms VPT [24] on most datasets (16 out of 19), and surpasses full fine-tuning along with other fine-tuning methods by large margins.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “There have been emerging a bunch of prompting methods, such as [35, 38, 26, 76, 77, 3, 24, 66, 14], to name a few 6 .”
CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “In Section 3.3, we provide several variants of CLIP-Adapter.”
Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, methodology): “Among these, prompt tuning employs learnable perturbations in the embedding space or pixel space (Jia et al., 2022; Bahng et al., 2022).”
Personalize Segment Anything Model with One Shot (2023, methodology): “Prompt Tuning [13, 22, 29, 62] suggests using learnable soft prompts alongside frozen models to perform specific downstream tasks, achieving more competitive performance with scale and robust domain transfer compared to full model tuning.”
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “Our bias tuning strategy bears similarity to prior parameter-efficient methodologies such as BitFit [70] for BERT fine-tuning and SSF [36] for visual prompt tuning [28].”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “…and down-projection is inserted into the transformer and only the parameters of this new module are updated; iv) Bias [66], where all the bias terms of parameters are updated; v) VPT [29], where the prompts are inserted into transformers as the input tokens and they are updated at fine-tuning.”
InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning (2024, background): “Through (2), we can ﬁnd that each column of ∂ L ∂ W can be represented as input h multiplied by a real value a k ( 1 ≤ k ≤ d O ). of Therefore, in the linear layer, each column the gradient ∂ L ∂ W lies in the span of input.”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, background): “Due to its efficiency in terms of parameters and convergence rate, prompt learning is found to be of great interest for adapting foundational models like CLIP for vision [21, 57, 45, 46] and vision-language tasks [59, 58, 61, 7].”
Steering Prototypes with Prompt-tuning for Rehearsal-free Continual Learning (2023, methodology): “In response, prompttuning, initially gaining traction in the NLP domain [31, 28] followed by the adaption to the visual domain [22], has been explored.”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “In the recent era of foundation models, a new paradigm, “pretraining and prompting” [59], [66], [67], [68], [69], [70], has emerged.”
MaPLe: Multi-modal Prompt Learning (2022, methodology): “Similar to [16] our design also uses deep ‘vision’ prompting.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, methodology): “Also, some prompt tuning schemes are designed to fix the parameters of the large model and adjust the parameters as little as possible to achieve good results, such as the VPT [263], CoOp [264], CoCoOp [265].”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “(6) Prompt Tuning [32]: Prepending a sequence of learnable prompt tokens to the input visual patch tokens.”
Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (2024, background): “Thus, several parameter-efficient fine-tuning methods [27, 28, 32, 35, 75] have been explored, which only set a few trainable parameters and fine-tune them for efficiency.”
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, neutral): “Later, prompting was brought to the vision community [24,31,34,59,74,75].”