Bottleneck Transformers for Visual Recognition

By A. Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani. 2021

1,200 citations recorded by the source corpus.

We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt [67] evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 1.64x faster in "compute"1 time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision.2

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, background): “Unlike the recent hybrid architectures (e.g., Hybrid-ViT [14] and BoTNet [45]) that rely on convolutions for feature encoding, Outlooker proposes to use local pair-wise token similarities to encode fine-level features and spatial context into tokens features and hence is more effective and parameter”
Perceiver: General Perception with Iterative Attention (2021, background): “Nevertheless they have been quickly percolating into perception, often as pieces of otherwise convolutional models, such as for images (Bello et al., 2019; Cordonnier et al., 2020; Srinivas et al., 2021) and videos (Wang et al.”
Scaling Vision Transformers (2021, methodology): “Hybrids of CNNs and Transformers typically allow smaller models to perform well, such as the ViT-CNN hybrid in [15], BoTNet [35], and HaloNet [43].”
High-Performance Large-Scale Image Recognition Without Normalization (2021, methodology): “2Code available at https://github.com/deepmind/ deepmind-research/tree/master/nfnets 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 Training Latency (s/step) on TPUv3, Batch Size per Device = 32 80 81 82 83 84 85 86 87 Im ag eN et T op -1 A cc ur ac y (% ) F0 F1 F2 F3 F4 NFNet-F5 LambdaNet-152 LambdaNet-420 EffNet”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “The selfattention layers can complement backbones [61, 7, 3, 65, 21, 68, 51] or head networks [29, 24] by providing the capability to encode distant dependencies or heterogeneous interactions.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “[29] carefully combines a standard convolutional backbone with bottleneck blocks based on self-attention instead of convolutions.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “A common approach is to augment the ConvNet backbone with explicit self-attention or non-local modules [9, 10, 11, 12], or to replace certain convolution layers with standard self-attention [11] or a more flexible mix of linear attention and convolution [41].”
MLP-Mixer: An all-MLP Architecture for Vision (2021, methodology): “[42] replace 3×3 convolutions in ResNets by self-attention layers.”
Going deeper with Image Transformers (2021, methodology): “On ImageNet1k-val [55], our model is on par with the state of the art (86.5”
CvT: Introducing Convolutions to Vision Transformers (2021, background): “Recently, BoTNet [27] proposes a simple yet powerful backbone architecture that just replaces the spatial convolutions with global self-attention in the ﬁnal three bottleneck blocks of a ResNet and achieves a strong performance in image recognition.”
ViViT: A Video Vision Transformer (2021, background): “Although self-attention has been employed extensively in computer vision, it has, in contrast, been typically incorporated as a layer at the end or in the later stages of the network [75, 7, 32, 77, 83] or to augment residual blocks [30, 6, 9, 57] within a ResNet architecture [27].”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, methodology): “Srinivas et al. (2021), rather than concatenating convolutional feature maps, propose to use a stride of 1 in the last stage of the ResNet architecture for improved performance.”
A ConvNet for the 2020s (2022, methodology): “Prior to ViT, the focus was on augmenting a ConvNet with self-attention/non-local modules [8, 55, 66, 79] to capture long-range dependencies.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, neutral): “Several recent papers [2, 39, 10, 62, 46] have attempted using self-attention primitives to improve image classification accuracy over the strong and commonly used ResNet backbones [14, 15].”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, methodology): “There has also been a lot of interest in combining CNNs with different forms of self-attention [2, 32, 48, 31, 3, 17, 39].”
ConViT: improving vision transformers with soft convolutional inductive biases (2021, neutral): “These models, which interleave or combine convolutional and self-attention layers, have fueled successful results in a variety of tasks [22, 23, 19, 24, 25, 26, 27, 28].”
EfficientNetV2: Smaller Models and Faster Training (2021, background): “, 2021) aim to improve training efficiency by removing the expensive batch normalization; Several recent works (Srinivas et al., 2021) focus on improving training speed by adding attention layers into convolutional networks (ConvNets); Vision Transformers (Dosovitskiy et al.”
DeepViT: Towards Deeper Vision Transformer (2021, background): “The recent progress of deep CNN models is largely driven by training very deep models with a large number of layers which is enabled by novel model architecture designs [9, 41, 31, 20, 47].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Bottleneck transformers [46] and “Visual Transformers” [47] (not to be confused with ViT) are both generic architectures that can also be used for detection and object segmentation.”