Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning

By Dongze Lian, Daquan Zhou, Jiashi Feng, Xinchao Wang. Neural Information Processing Systems, 2022

390 citations recorded by the source corpus.

Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, we propose a new parameter-efficient fine-tuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine-tuning. In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters. Furthermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MLPs) and datasets. Results on 26 image classification datasets in total and 3 robustness&out-of-distribution datasets show the effectiveness of SSF. Code is available at https://github.com/dongzelian/SSF.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/f0b31fdf53ad60df454afd4ec8633b3aeb347bff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peng Gao
2. Hongsheng Li
3. Yu Qiao
4. Renrui Zhang
5. Jiaming Han
6. Pan Lu
7. Shijie Geng
8. W. Zhang

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Peng Gao
3. Yu Qiao
4. Renrui Zhang
5. Pan Lu
6. Xiaogang Wang
7. Jiaming Han
8. Luc Van Gool

In Radiology, Nuclear Medicine and Imaging

1. Yu Qiao
2. Xiangyu Yue
3. Hongsheng Li

Citations in the corpus

• PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning (2024, background): “BLIP [16] marked a significant milestone by integrating a frozen vision encoder with BLIP to enhance video processing efficiency, with only the newly added Q-Former learnable.”
• LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model (2023, methodology): “Our bias tuning strategy bears similarity to prior parameter-efficient methodologies such as BitFit [70] for BERT fine-tuning and SSF [36] for visual prompt tuning [28].”
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters (2021, neutral): “Some other methods, e.g., black-box tuning [56], ladder side-tuning [57], sparse structure search[26], and Scaling&Shifting [38] Prompt Design.”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “…v2 [121], APT [224], p-tuning [123], prompt-tuning [102], Xprompt [126], IDPG [189], LPT [122], SPT [240], APrompt [177] Training Speedup SPoT [173], TPT [166], InfoPrompt [185], PTP [18], IPT [148], SMoP [27], DePT [160] Others (IA) 3 [116], MoV [206], SSF [109], IPA [124] SelectiveFine-tuning”
• EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “In Sections IV-B and IV-C, we will delve into the detailed processes involved in constructing various visual task-oriented multiturn conversation data, including coarse-grained and fine-grained conversation scenarios.”