Toward Transformer-Based Object Detection

By Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, Dmitry Kislyuk. 2020

257 citations recorded by the source corpus.

Transformers have become the dominant model in natural language processing, owing to their ability to pretrain on massive amounts of data, then transfer to smaller, more specific tasks via fine-tuning. The Vision Transformer was the first major attempt to apply a pure transformer model directly to images as input, demonstrating that as compared to convolutional networks, transformer-based architectures can achieve competitive results on benchmark classification tasks. However, the computational complexity of the attention operator means that we are limited to low-resolution inputs. For more complex tasks such as detection or segmentation, maintaining a high input resolution is crucial to ensure that models can properly identify and reflect fine details in their output. This naturally raises the question of whether or not transformer-based architectures such as the Vision Transformer are capable of performing tasks other than classification. In this paper, we determine that Vision Transformers can be used as a backbone by a common detection task head to produce competitive COCO results. The model that we propose, ViT-FRCNN, demonstrates several known properties associated with transformers, including large pretraining capacity and fast fine-tuning performance. We also investigate improvements over a standard detection backbone, including superior performance on out-of-domain images, better performance on large objects, and a lessened reliance on non-maximum suppression. We view ViT-FRCNN as an important stepping stone toward a pure-transformer solution of complex vision tasks such as object detection.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Benchmarking Detection Transfer Learning with Vision Transformers (2021, neutral): “[2] integrate standard ViT models with Faster R-CNN [34], but report substantially lower AP box compared to our results ( > 10 points lower), which suggests that our design is highly effective.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, neutral): “There are a few works applying ViT models to the dense vision tasks of object detection and semantic segmentation by direct upsampling or deconvolution but with relatively lower performance [2, 73].”
Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [115, 99], 3D reconstruction [72], pose estimation [107], generative modeling [14], image retrieval [27], medical image segmentation [13, 97, 111], point clouds [40], video in”
Transformer in Transformer (2021, methodology): “This design also affects the subsequent works including DeiT [31], ViT-FRCNN [2], IPT [6] and SETR [38].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, methodology): “The vision transformer has been thereafter successfully adapted for a wider range of computer vision tasks including object detection [30], semantic segmentation [31] and image retrieval [32].”
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training (2021, methodology): “Inspired by ViT, there are some recent works studying vision Transformer for a broad range of vision tasks such as image classification [31, 39], object detection [4], semantic segmentation [45].”
A Survey on Vision Transformer (2020, methodology): “Transformers lack some inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.”
On the Integration of Self-Attention and Convolution (2021, methodology): “Furthermore, transformer paradigm is adopted in detection [2,7,56], segmentation [45,52,54], point cloud recognition [17, 32] and other vision tasks [8, 34].”
UniT: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[40, 3, 22, 21, 47, 15, 61, 5, 72, 2, 58]).”
Blending Anti-Aliasing into Vision Transformer (2021, background): “After a big leap made by vision transformer (ViT) [1], promising results on vision tasks have recently emerged [2–4], demonstrating a possibility of using transformers as a primary backbone for vision applications.”
Inception Transformer (2022, background): “Later, many ViT variants [5–13] have been developed to boost performance or scale to a wider range of vision tasks, e.g. , object detection [10, 11] and segmentation [12, 13].”
Conformer: Local Features Coupling Global Representations for Visual Recognition (2021, methodology): “Recently, the transformer architecture [42] has been introduced to visual tasks [16, 47, 41, 51, 8, 9, 3, 55, 28].”
You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection (2021, methodology): “ViT-FRCNN [6] is the ﬁrst to use a pre-trained ViT as the backbone for a Faster R-CNN [50] object detector.”
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation (2021, background): “Initially people try to directly learn high-resolution features extracted by ViT back-bone via extra interpolation or convolution layers [2, 46].”
TransReID: Transformer-based Object Re-Identification (2021, background): “ViT-FRCNN [2] shows that the output embeddings of the last layer can be reshaped as a spatial feature map that includes location information.”
Fine-tuning Image Transformers using Learnable Memory (2022, background): “This fine-tuning has been shown to help achieve high performance on a variety of downstream tasks, from classical tasks like image super-resolution [6] and object detection [3] to novel ones, such as Ising model simulations [19].”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, background): “More recently Vision Transformers (ViT) [17] have shown promising performance and are rivaling convolutional neural networks (CNN) and a wide range of modifications have recently been proposed to apply them to different vision tasks [1,2,21,55,68,73,78,90].”
Multimodal Learning With Transformers: A Survey (2022, unknown): “Both ViT and its variants have been widely applied to various computer vision tasks, including low-level tasks [92], recognition [93], detection [94], segmentation [95], etc, and also work well for both supervised [93] and self-supervised [96], [97], [98] visual learning.”