Exploring Self-Attention for Image Recognition

By Hengshuang Zhao, Jiaya Jia, Vladlen Koltun. 2020

932 citations recorded by the source corpus.

Recent work has shown that self-attention can serve as a basic building block for image recognition models. We explore variations of self-attention and assess their effectiveness for image recognition. We consider two forms of self-attention. One is pairwise self-attention, which generalizes standard dot-product attention and is fundamentally a set operator. The other is patchwise self-attention, which is strictly more powerful than convolution. Our pairwise self-attention networks match or outperform their convolutional counterparts, and the patchwise models substantially outperform the convolutional baselines. We also conduct experiments that probe the robustness of learned representations and conclude that self-attention networks may have significant benefits in terms of robustness and generalization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “eighborhoods for each query pixel instead of globally. Such local multi-head dot-product self attention blocks can completely replace convolutions (Ramachandran et al., 2019; Cordonnier et al., 2020; Zhao et al., 2020). Alternatively, works such as Sparse Transformers (Child et al., 2019) employ sca”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “[18, 13, 21, 17, 15] on the other hand restrict receptive field of self-attention and use it in a convolutional manner.”
Early Convolutions Help Transformers See Better (2021, background): “[53] explore a broader set of self-attention operations with hard-coded locality constraints, more similar to standard CNNs.”
Emerging Properties in Self-Supervised Vision Transformers (2021, background): “Transformers [70] have recently emerged as an alternative to convolutional neural networks (convnets) for visual recognition [19, 69, 83].”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, background): “There have been continuous efforts on generalizing Transformers to computer vision [43, 5, 38, 48, 6, 15].”
Perceiver: General Perception with Iterative Attention (2021, methodology): “, 2020a); others replace convolution with patchwise self-attention (Parmar et al., 2018; Ramachandran et al., 2019; Zhao et al., 2020).”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Our Swin-T architecture brings consistent +3.4∼4.2 box AP gains over ResNet-50, with slightly larger model size, FLOPs and latency.”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “These are in fact open questions for self-attention in general (Xiong et al., 2020; Liu et al., 2020; Zhao et al., 2020) and developments in this direction are of utmost importance.”
Transformers in Vision: A Survey (2021, background): “attention from [72] is a nice work in this direction which attempts to specifically tailor self-attention operation for visual inputs.”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, background): “SAN [59] explores two types of self-attention modules.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, background): “More recently, SAN [53] explores both the pairwise and patchwise self-attention for image recognition, where the patchwise self-attention is a generalization Reshape”
Going deeper with Image Transformers (2021, methodology): “Various other attention mechanism have been used successfully to give a global view in conjunction with (local) convolutions [4, 52, 12, 77, 79], most mimic squeeze-and-excitate [32] for leveraging global features.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Recent studies showed that convolution-free, specially designed all-attention models can match CNNs’ performance on image recognition tasks [112, 48, 79].”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “This broadens the family of functions they can represent (Cordonnier et al., 2020; Zhao et al., 2020), and renders them better suited to modern big-data regimes where there is less need for strong inductive priors.”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “0) Local relative self-attention (Zhao et al., 2020) 20.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “Self-attention has several properties that make it a good fit for vision: (a) content-based interactions as opposed to content-independent interactions of convolution; (b) parameter-independent scaling of receptive field size as opposed to parameter-dependent scaling of convolution; (c) empirical ab”