AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

By Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, Ping Luo. Neural Information Processing Systems, 2022

1,174 citations recorded by the source corpus.

Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains. To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently. It possesses several benefits more appealing than prior arts. Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks. Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks. Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2fe2f849b94cf08b559226bc9d78adcaef5ef186

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Han-Jia Ye
2. De-chuan Zhan
3. Da-Wei Zhou
4. Hai-Long Sun
5. Daquan Zhou
6. Jiashi Feng
7. Xinchao Wang
8. Chen Zhao

In Computer Vision and Pattern Recognition

1. Han-Jia Ye
2. De-chuan Zhan
3. Jiashi Feng
4. Da-Wei Zhou
5. Huchuan Lu
6. Xinchao Wang
7. Dong Wang
8. Ming-Hsuan Yang

Citations in the corpus

• InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning (2024, background): “…of the subspace M ⊥ t in different Transformer layers of ViT-B/16. find We can that the dimension of the subspace M ⊥ t in different Transformer layers of ViT-B/16 is always much larger than zero, which means the space for learning the new task always exists throughout the whole learning process.”
• FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification (2022, neutral): “, 2022), AdaptFormer (Chen et al., 2022), NOAH (Zhang et al.”
• Visual Prompt Multi-Modal Tracking (2023, background): “AdaptFormer [4] introduces lightweight modules to ViT and outperforms full fine-tuned models on action recognition benchmarks.”
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “During the submission of this work, some methods [3, 68] are also proposed for parameter-efficient fine-tuning, e.g. , inserting a adapter module or neural prompt search.”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “A variety of methods leverage PEFT techniques to achieve efficient model tuning [84], [25], [21], [86].”
• Improving Visual Prompt Tuning for Self-supervised Vision Transformers (2023, methodology): “Chen et al. (2022) introduced an additional lightweight module, known as AdaptMLP, in an MLP module of ViT blocks and fine-tune it.”
• A Unified Continual Learning Framework with General Parameter-Efficient Tuning (2023, positive): “In addition to the three PET modules described above, there are many others, such as AdaptBias [7], Com-pacter [33], and AdapterFormer [3], and there will be new and superior PET methods in future as well.”
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation (2023, neutral): “C V ] 29 D ec 2023 conditioned adaptation, in which we use the visual prompt to generate a series of weights that can be applied to the adaptation embedding efficiently, facilitating wide and deep prompt-adaptation interactions.”
• Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning (2024, methodology): “Following [8], we learn an adapter module as a side branch for the MLP. Specifically, an adapter is a bottleneck module that contains a down-projection layer W down ∈ R d × r , a non-linear activation function σ , and an up-projection layer W up ∈ R r × d .”