A2-Nets: Double Attention Networks

By Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, Jiashi Feng. 2018

608 citations recorded by the source corpus.

Learning to capture long-range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block", a novel component that aggregates and propagates informative global features from the entire spatio-temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second-order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet-50 equipped with our double attention blocks outperforms a much larger ResNet-152 architecture on ImageNet-1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state-of-the-art results on the Kinetics and UCF-101 datasets with significantly higher efficiency than recent works.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, neutral): “We compare our results with other previous methods [7, 9, 16, 38, 15, 18, 17] that adopt attention in different ways on ResNets.”
Stand-Alone Self-Attention in Vision Models (2019, neutral): “We achieve this in two steps:”
Attention Augmented Convolutional Networks (2019, background): “Additionally, multiple attention mechanisms have been proposed for visual tasks to address the weaknesses of convolutions [17, 16, 7, 46, 45, 53].”
VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “vision task [22], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, methodology): “Self-attention in CNN Self-attention mechanism has been widely applied to CNN in vision task [38, 55, 19, 47, 20, 39, 1, 6, 18, 31, 42, 13].”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “However, we note that Trans-formers have been adopted on top of convolutional feature maps for action localization and recognition (Girdhar et al., 2019), video classiﬁcation (Wang et al., 2018b; Chen et al., 2018b), and group activity recognition (Gavrilyuk et al., 2020).”
ViViT: A Video Vision Transformer (2021, background): “Although self-attention has been employed extensively in computer vision, it has, in contrast, been typically incorporated as a layer at the end or in the later stages of the network [75, 7, 32, 77, 83] or to augment residual blocks [30, 6, 9, 57] within a ResNet architecture [27].”
ConViT: improving vision transformers with soft convolutional inductive biases (2021, background): “Chen et al. (2018) and Bello et al. (2019) leveraged this complementarity by augmenting convolutional layers with attention.”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “Recently, the self-attention module [57] has been explored by several works [13, 18, 23, 26, 60] for computer vision tasks.”
Incorporating Convolution Designs into Visual Transformers (2021, methodology): “To utilize the advantages of self-attention in building longrange dependencies, some work introduces attention modules into CNNs [49, 45, 40, 3, 8, 17, 42].”
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, methodology): “Since then, improvements to this paradigm include the use of ImageNet-inflated weights [11], the space-time decomposition of 3D convolutions [54, 74, 84], channel-separated convolutions [73], non-local blocks [79], and attention layers [13].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Motivated by this, several works [18], [114] treat it as a fully-connected graph or a non-local module for visual recognition task.”