Improving Visual Prompt Tuning for Self-supervised Vision Transformers

By S. Yoo, Eunji Kim, Dahuin Jung, Jungbeom Lee, Sung-Hoon Yoon. 2023

81 citations recorded by the source corpus.

Visual Prompt Tuning (VPT) is an effective tuning method for adapting pretrained Vision Transformers (ViTs) to downstream tasks. It leverages extra learnable tokens, known as prompts, which steer the frozen pretrained ViTs. Although VPT has demonstrated its applicability with supervised vision transformers, it often underperforms with self-supervised ones. Through empirical observations, we deduce that the effectiveness of VPT hinges largely on the ViT blocks with which the prompt tokens interact. Specifically, VPT shows improved performance on image classification tasks for MAE and MoCo v3 when the prompt tokens are inserted into later blocks rather than the first block. These observations suggest that there exists an optimal location of blocks for the insertion of prompt tokens. Unfortunately, identifying the optimal blocks for prompts within each self-supervised ViT for diverse future scenarios is a costly process. To mitigate this problem, we propose a simple yet effective method that learns a gate for each ViT block to adjust its intervention into the prompt tokens. With our method, prompt tokens are selectively influenced by blocks that require steering for task adaptation. Our method outperforms VPT variants in FGVC and VTAB image classification and ADE20K semantic segmentation. The code is available at https://github.com/ryongithub/GatedPromptTuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Jiaming Han
6. Pan Lu
7. Xiangfei Hu
8. Shilin Yan

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Peng Gao
3. Yu Qiao
4. Renrui Zhang
5. Pan Lu
6. Xiaogang Wang
7. Jiaming Han
8. Luc Van Gool

In Representation & Self-Supervised Learning

1. Yu Qiao
2. Peng Gao
3. Renrui Zhang
4. Hongsheng Li
5. Aojun Zhou

Citations in the corpus

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Moreover, we verify the effectiveness of our approach to fine-tune large models in other domains.”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “The work by [203] notices that compared with supervised ViT, VPT often underperforms with self-supervised ViT.”