Vision Transformer Adapter for Dense Predictions

By Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, Yu Qiao. 2022

888 citations recorded by the source corpus.

This work investigates a simple yet powerful dense prediction task adapter for Vision Transformer (ViT). Unlike recently advanced variants that incorporate vision-specific inductive biases into their architectures, the plain ViT suffers inferior performance on dense predictions due to weak prior assumptions. To address this issue, we propose the ViT-Adapter, which allows plain ViT to achieve comparable performance to vision-specific transformers. Specifically, the backbone in our framework is a plain ViT that can learn powerful representations from large-scale multi-modal data. When transferring to downstream tasks, a pre-training-free adapter is used to introduce the image-related inductive biases into the model, making it suitable for these tasks. We verify ViT-Adapter on multiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Notably, without using extra detection data, our ViT-Adapter-L yields state-of-the-art 60.9 box AP and 53.0 mask AP on COCO test-dev. We hope that the ViT-Adapter could serve as an alternative for vision-specific transformers and facilitate future research. The code and models will be released at https://github.com/czczup/ViT-Adapter.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “Various PEFT techniques have been explored, including prompt tuning [35, 30, 39, 38, 50, 72], Low-Rank Adaptation (LoRA) [23, 69, 20], and adapters [22, 48, 37, 9, 55].”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Models for both tasks have evolved from CNN-based architectures [48] to transformer-based ones [11], and from two-stage models [28], one-stage models [3, 66] to the recent query-based approaches [18,92].”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, background): “ViT and its variants [13, 23, 30, 71, 89, 111, 112, 125, 139, 140] have significantly increased their capacity and excelled in various important visual tasks.”
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models (2023, methodology): “Recently, ViT-Adapter [3] utilized adapters to enable a plain ViT to perform different downstream tasks.”
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (2024, neutral): “Segmenter [57] ViT-L [16] 51.8 SegFormer [69] MiT-B5 [69] 51.0 Mask2Former [12] Swin-L [38] 56.4 UperNet [68] BEiT-L [2] 56.3 ViT-Adapter [11] BEiT-L [2] 58.3 OneFormer [24] Swin-L [38] 57.4 OneFormer [24] ConNeXt-XL [40] 57.4 Ours ViT-L [16] 59.4 Table 8.”
Focal Modulation Networks (2022, neutral): “1 ViT-Adapter-L† [11] Mask2Former [12] 568M 58.”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, background): “Work on this problem has led to significant improvements along several threads including image level predictions like image classification [13, 54, 58], object level predictions like object detection [33,51,53,59–61], as well as pixel level predictions like semantic segmentation [10,29,51,53].”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, neutral): “ViT-Adapter (Chen et al., 2023) and ViT-CoMer (Xia et al., 2024) supplement a pretrained ViT with a convolutional network for dense prediction, whereas ViT-Det (Li et al., 2022b) builds a feature pyramid on top of a pretrained ViT.”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Following the idea of learnable tuning, recent works [268], [269] extend the vision adapter into dense prediction tasks, including segmentation and detection.”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, neutral): “The traditional paradigm of vision foundation models is pre-training followed by fine-tuning [51, 11, 43, 53, 17, 44], which is effective but comes with significant marginal costs when adapting to diverse downstream scenarios.”
EVA-02: A Visual Representation for Neon Genesis (2023, neutral): “6 & 9 numerical precision PyTorch amp fp16 [91] ViT-Adapter [27] ✗”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “More recently, adapters have yielded successful results in vision transformers [49, 50] and ViT-Adapter [14].”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “For reference, using the Mask2Former pipeline (Steiner et al., 2021) with a ViT-Adapter (Chen et al., 2023b) on top of our frozen ViT-g/14 backbone gives 60.2 mIoU on ADE-20k.”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, methodology): “To further reach the leading performance by leveraging our model, we arm InternImage-H with more advanced segmenter Mask2Former [76], and adopt the same training settings in [17, 66].”