Stand-Alone Self-Attention in Vision Models

By Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jonathon Shlens. 2019

1,375 citations recorded by the source corpus.

Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependencies. These efforts focus on augmenting convolutional models with content-based interactions, such as self-attention and non-local means, to achieve gains on a number of vision tasks. The natural question that arises is whether attention can be a stand-alone primitive for vision models instead of serving as just an augmentation on top of convolutions. In developing and testing a pure self-attention vision model, we verify that self-attention can indeed be an effective stand-alone layer. A simple procedure of replacing all instances of spatial convolutions with a form of self-attention applied to ResNet model produces a fully self-attentional model that outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters. On COCO object detection, a pure self-attention model matches the mAP of a baseline RetinaNet while having 39% fewer FLOPS and 34% fewer parameters. Detailed ablation studies demonstrate that self-attention is especially impactful when used in later layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Inspired by NLP successes, multiple works try combining CNN-like architectures with self-attention (Wang et al., 2018; Carion et al., 2020), some replacing the convolutions entirely (Ramachandran et al., 2019; Wang et al., 2020a).”
Experience Grounds Language (2020, background): “ality, these architectures are mature and capable of generalizing, both from the perspective of engineering infrastructure3 and that the backbones have been tested against other technologies like self-attention (Ramachandran et al., 2019).”
Reformer: The Efficient Transformer (2020, methodology): “The Transformer model introduced in (Vaswani et al., 2017) has been used widely in natural language tasks and further extended to model diverse data such as music scores (Huang et al., 2018), and images (Parmar et al., 2018; Ramachandran et al., 2019).”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “[18, 13, 21, 17, 15] on the other hand restrict receptive field of self-attention and use it in a convolutional manner.”
Exploring Self-Attention for Image Recognition (2020, background): “We also explore a family of patchwise attention operators that are structurally different from the forms used in [13, 27] and constitute strict generalizations of convolution.”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “In particular, on ImageNet, we build an Axial-ResNet by replacing the 3 × 3 convolution in all residual blocks [32] with our positionsensitive axial-attention layer, and we further make it fully attentional [68] by adopting axial-attention layers in the ‘stem’.”
On the Relationship between Self-Attention and Convolutional Layers (2019, methodology): “It is only recently that Bello et al. (2019); Ramachandran et al. (2019) used transformers on images and showed that they achieve similar accuracy as ResNets.”
Attention Augmented Convolutional Networks (2019, background): “For instance, identifying a local attention mechanism may result in an efficient and scalable computational mechanism that could prevent the need for downsampling with average pooling [34].”
Early Convolutions Help Transformers See Better (2021, background): “[32] explore this direction further with stand-alone self-attention networks for vision.”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “Besides modeling global attention, there are also some works [36], [37] exploring how to exploit self-attention in local patches to reduce the memory and computation cost.”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, background): “There have been continuous efforts on generalizing Transformers to computer vision [43, 5, 38, 48, 6, 16].”
Perceiver: General Perception with Iterative Attention (2021, methodology): “, 2020a); others replace convolution with patchwise self-attention (Parmar et al., 2018; Ramachandran et al., 2019; Zhao et al., 2020).”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, background): “Since then, attention-based architectures have permeated across data domains machine learning applications, such as in natural language processing [DCLT18, POT16], speech recognition [LZL + 20], and computer vision [RPV + 19, BZV + 19].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Works that use only self-attention blocks tend to apply them at an individual pixel level [16, 46, 50], often requiring tricks to ensure computational tractability, including re-stricting the scope of self-attention to a local neighbourhood [50], adding global self-attention on heavily down-sized…”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, neutral): “Most works (Liu et al., 2018a; Parmar et al., 2018; Child et al., 2019; Huang et al., 2019; Ho et al., 2019; Hu et al., 2019; Parmar et al., 2019; Qiu et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) follow this paradigm.”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, background): “While some prior arts have incorporated the attention modules [53, 35, 61] into CNNs, as far as we know, exploring a clean and convolution-free Transformer backbone to address dense prediction tasks in computer vision is rarely studied.”
Efficient Transformers: A Survey (2020, background): “Limiting the receptive field to a local neighborhood [54, 55] addresses the issues with the computational and memory costs of running global self-attention on large inputs, but changing the neighborhood per query position would prohibit packing the computations of the self-attention into two matrix ”