Attention Augmented Convolutional Networks

By Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V. Le. 2019

1,167 citations recorded by the source corpus.

Convolutional networks have enjoyed much success in many computer vision applications. The convolution operation however has a significant weakness in that it only operates on a local neighbourhood, thus missing global information. Self-attention, on the other hand, has emerged as a recent advance to capture long range interactions, but has mostly been applied to sequence modeling and generative modeling tasks. In this paper, we propose to augment convolutional networks with self-attention by concatenating convolutional feature maps with a set of feature maps produced via a novel relative self-attention mechanism. In particular, we extend previous work on relative self-attention over sequences to images and discuss a memory efficient implementation. Unlike Squeeze-and-Excitation, which performs attention over the channels and ignores spatial information, our self-attention mechanism attends jointly to both features and spatial locations while preserving translation equivariance. We find that Attention Augmentation leads to consistent improvements in image classification on ImageNet and object detection on COCO across many different models and scales, including ResNets and a state-of-the art mobile constrained network, while keeping the number of parameters similar. In particular, our method achieves a 1.3% top-1 accuracy improvement on ImageNet classification over a ResNet50 baseline and outperforms other attention mechanisms for images such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 AP in COCO Object Detection on top of a RetinaNet baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “n hardware accelerators. There has also been a lot of interest in combining convolutional neural networks (CNNs) with forms of self-attention, e.g. by augmenting feature maps for image classiﬁcation (Bello et al., 2019) or by further processing the output of a CNN using self-attention, e.g. for obje”
End-to-End Object Detection with Transformers (2020, methodology): “Since the transformer architecture is permutation-invariant, we supplement it with fixed positional encodings [31,3] that are added to the input of each attention layer.”
Rethinking Attention with Performers (2020, background): “Several solutions have been proposed to address this issue (Beltagy et al., 2020; Gulati et al., 2020; Chan et al., 2020; Child et al., 2019; Bello et al., 2019).”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “[16] use self-attention to augment convolutions and and reduce the compute cost by using small channel sizes for attention.”
Exploring Self-Attention for Image Recognition (2020, background): “[1] combined convolutional and self-attention processing streams, but found that the global self-attention they used was not sufficiently powerful to replace convolutions entirely.”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Attention has since been extended to vision, giving significant boosts to image classification [6], object detection [37], semantic segmentation [42], video classification [87], and adversarial defense [89].”
On the Relationship between Self-Attention and Convolutional Layers (2019, methodology): “In practice, it was shown by Bello et al. (2019) that combining CNN and self-attention features outperforms each taken separately.”
Stand-Alone Self-Attention in Vision Models (2019, neutral): “We achieve this in two steps:”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “vision task [22], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41].”
Perceiver: General Perception with Iterative Attention (2021, background): “Nevertheless they have been quickly percolating into perception, often as pieces of otherwise convolutional models, such as for images (Bello et al., 2019; Cordonnier et al., 2020; Srinivas et al., 2021) and videos (Wang et al.”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, methodology): “Since then, attention-based architectures have permeated across data domains machine learning applications, such as in natural language processing [DCLT18, POT16], speech recognition [LZL + 20], and computer vision [RPV + 19, BZV + 19].”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, neutral): “self-attention improves over using them individually [14].”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, positive): “AANet [3] achieves competitive results when combining the self-attention and convolutional operations.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “More recently, it has been proposed to replace CNNs with self-attention networks either partially (Bello et al., 2019) or entirely (Ramachandran et al., 2019).”
Transformers in Vision: A Survey (2021, background): “Exist- 5 Local Relation Networks Hu et al. 2019 Stand-Alone Attention Ramachandran et al. 2019 Local Attention Transformer Global Attenion Criss-Cross Attention Haung et al. 2019 Non-Local Networks Buades et al. 2018 ViT Dosovitskiv et al. 2020 Color Transformer 2020 Pair/Patch Attention Zhao et al.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “A common approach is to augment the ConvNet backbone with explicit self-attention or non-local modules [9, 10, 11, 12], or to replace certain convolution layers with standard self-attention [11] or a more flexible mix of linear attention and convolution [41].”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, background): “AANet [2] mixes convolution and self-attention for backbone training.”