Exploring Plain Vision Transformer Backbones for Object Detection

By Yanghao Li, Hanzi Mao, Ross B. Girshick, Kaiming He. 2022

1,190 citations recorded by the source corpus.

We explore the plain, non-hierarchical Vision Transformer (ViT) as a backbone network for object detection. This design enables the original ViT architecture to be fine-tuned for object detection without needing to redesign a hierarchical backbone for pre-training. With minimal adaptations for fine-tuning, our plain-backbone detector can achieve competitive results. Surprisingly, we observe: (i) it is sufficient to build a simple feature pyramid from a single-scale feature map (without the common FPN design) and (ii) it is sufficient to use window attention (without shifting) aided with very few cross-window propagation blocks. With plain ViT backbones pre-trained as Masked Autoencoders (MAE), our detector, named ViTDet, can compete with the previous leading methods that were all based on hierarchical backbones, reaching up to 61.3 AP_box on the COCO dataset using only ImageNet-1K pre-training. We hope our study will draw attention to research on plain-backbone detectors. Code for ViTDet is available in Detectron2.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “The effectiveness of this simple feature pyramid is first demonstrated in ViTDet [23] for object detection.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, background): “gnConv fully inherits the translation equivariance of the standard convolution, which introduces beneficial inductive biases to major vision tasks and avoids the asymmetry brought by local attention [40, 33].”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “Given any image, we first use an off-the-shelf detector [39] and a body keypoints estimator [78] to get bounding boxes and corresponding 2D keypoints.”
EVA-02: A Visual Representation for Neon Genesis (2023, methodology): “Table 28: COCO object detection and instance segmentation, headto-head comparisons setting based on ViTDet [75].”
BiFormer: Vision Transformer with Bi-Level Routing Attention (2023, methodology): “Recently, to take advantage of large-scale pretraining with masked image modeling, a new research direction emerges to adapt plain ViT [2] for dense prediction tasks [1, 3].”
ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, methodology): “8 AP, which is comparable to the strategy proposed in ViTDet [26] that jointly uses full and window attention (the 6th row) but has much lower memory footprint, i.”
SAM-Adapter: Adapting Segment Anything in Underperformed Scenes (2023, background): “In the computer vision community, [30] suggested ﬁne-tuning the ViT [11] for object detection with minimal modiﬁcations.”
Scaling Open-Vocabulary Object Detection (2023, methodology): “OWLv2 therefore matches the performance of ViTDet with a Cascade Mask-RCNN head [18], despite using a simpler head architecture.”
DETRs with Hybrid Matching (2022, positive): “Notably, the very recent ViT-Det [25] models”
SAM 2: Segment Anything in Images and Videos (2024, methodology): “We follow Li et al. (2022b) and use global attention in only a subset of the image encoder layers (see Table 13).”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “More recently, adapters have yielded successful results in vision transformers [49, 50] and ViT-Adapter [14].”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “As we can see, when using regular training settings for fair comparison, the detection performance of ViT (Li et al., 2021b) and ViTDet (Li et al., 2022b) is inferior to recent vision-speciﬁc models.”
RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose (2023, methodology): “Transformer-based architectures [55] ported from modern Natural Language Processing (NLP) have achieved great success in various vision tasks like representation learning [15, 39], object detection [8, 34, 67], semantic segmentation [66], video understanding [4, 17, 40], as well as pose estimation [”
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks (2023, background): “Thanks to the strong universal representation learned by Florence-2 pre-training, we do not require longer training epochs, such as 36 epochs in [51, 81, 85, 86], or 100 epochs in [46], to achieve better results.”