Benchmarking Detection Transfer Learning with Vision Transformers

By Yanghao Li, Saining Xie, Xinlei Chen, Piotr Doll'ar, Kaiming He, Ross B. Girshick. 2021

211 citations recorded by the source corpus.

Object detection is a central downstream task used to test if pre-trained network parameters confer benefits, such as improved accuracy or training speed. The complexity of object detection methods can make this benchmarking non-trivial when new architectures, such as Vision Transformer (ViT) models, arrive. These difficulties (e.g., architectural incompatibility, slow training, high memory consumption, unknown training formulae, etc.) have prevented recent studies from benchmarking detection transfer learning with standard ViT models. In this paper, we present training techniques that overcome these challenges, enabling the use of standard ViT models as the backbone of Mask R-CNN. These tools facilitate the primary goal of our study: we compare five ViT initializations, including recent state-of-the-art self-supervised learning methods, supervised initialization, and a strong random initialization baseline. Our results show that recent masking-based unsupervised learning methods may, for the first time, provide convincing transfer learning improvements on COCO, increasing box AP up to 4% (absolute) over supervised and prior self-supervised pre-training methods. Moreover, these masking-based initializations scale better, with the improvement growing as model size increases.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [16,48], object detection [7,45], semantic segmentation [66,83], panoptic segmentation [73], video understanding [21] and fewshot learning [15], surpassing previous state-of-the-”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “2 This work is an extension of a preliminary version [33] that was unpublished and not submitted for peer review.”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “Therefore, we use a simpler evaluation procedure inspired by [49], which consists of resizing the image so that the largest side matches the training resolution and padding the smallest side.”
Universal Guidance for Diffusion Models (2023, methodology): “For this experiment, we use FasterRCNN [21] with Resnet-50-FPN backbone [15], a publicly”
Adding Conditional Control to Text-to-Image Diffusion Models (2023, background): “More recently, adapters have yielded successful results in vision transformers [49, 50] and ViT-Adapter [14].”
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking (2022, methodology): “We adopt the standard practice to extract single-scale features from different Transformer layers, such as layers 4, 6, 8, and 12 of the LayoutLMv3 base model, and use resolution-modifying modules to convert the single-scale features into the multiscale FPN features [1, 27, 30].”
A Unified Sequence Interface for Vision Tasks (2022, background): “Models with a shared backbone can learn general features which are transferable across tasks when scaling up with training tasks, model capacity and data [20, 24, 12].”
Vision Transformer Adapter for Dense Predictions (2022, methodology): ““∗” indicates following [43] to produce multi-scale feature maps with transposed convolutions.”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “Simple and free from dataset or task-specific augmentations [43], MAE is the state95 of-the-art self-supervised framework in computer vision [44, 45, 46, 47], and has been demonstrated 96 to work well for motor control tasks in simulation [9].”