Residual Attention Network for Image Classification

By Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, Xiaoou Tang. 2017

3,644 citations recorded by the source corpus.

In this work, we propose Residual Attention Network, a convolutional neural network using attention mechanism which can incorporate with state-of-art feed forward network architecture in an end-to-end training fashion. Our Residual Attention Network is built by stacking Attention Modules which generate attention-aware features. The attention-aware features from different modules change adaptively as layers going deeper. Inside each Attention Module, bottom-up top-down feedforward structure is used to unfold the feedforward and feedback attention process into a single feedforward process. Importantly, we propose attention residual learning to train very deep Residual Attention Networks which can be easily scaled up to hundreds of layers. Extensive analyses are conducted on CIFAR-10 and CIFAR-100 datasets to verify the effectiveness of every module mentioned above. Our Residual Attention Network achieves state-of-the-art object recognition performance on three benchmark datasets including CIFAR-10 (3.90% error), CIFAR-100 (20.45% error) and ImageNet (4.8% single model and single crop, top-5 error). Note that, our method achieves 0.6% top-1 accuracy improvement with 46% trunk depth and 69% forward FLOPs comparing to ResNet-200. The experiment also demonstrates that our network is robust against noisy labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring Self-Attention for Image Recognition (2020, background): “A number of methods learned to reweight convolutional activations or offset the taps of convolutional kernels [5, 15, 34, 36, 46], thus retaining the basic principle of convolutional feature construction.”
VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “vision task [22], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41].”
Dynamic Neural Networks: A Survey (2021, background): “Spatial locations in features could also be dynamically rescaled with attention to improve the representation power of deep models [128].”
Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks (2018, background): “Top-down attention modules [42] and self-attention [41] can be used to exploit global relationships between features.”
BAM: Bottleneck Attention Module (2018, background): “A few attempts [19, 42] have been made to consider attention as an effective solution for general classiﬁcation task.”
Global-and-local attention networks for visual recognition (2018, background): “This is the type of attention used in most state-of-the-art networks [12, 6, 13].”
GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (2020, background): “By dint of automatic feature engineering, deep neural networks (DNNs) have achieved remarkable success in various computer vision tasks, such as image classification [37, 36, 41, 33, 32, 15, 39], visual generation [34, 35], image retrieval [40, 7, 8, 12] and semantic comprehension [18, 17].”
CBAM: Convolutional Block Attention Module (2018, methodology): “[27] propose Residual Attention Network which uses an encoderdecoder style attention module.”
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents (2019, background): “image classification and captioning [12, 13, 14, 15, 16, 17, 18, 19, 20], text classification [21, 22], generative models [6, 23, 24], object tracking [25], and reinforcement learning [26, 27, 28].”
Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting (2019, background): “Attention mechanisms are used in translation [36], image classification [37] or tabular learning [3] to identify salient portions for a specific example – using the magnitude of attention weights to determine the importance of different locations.”
Pre-Trained Models: Past, Present and Future (2021, background): “…Wu et al., 2016), graph neural networks (GNNs) (Kipf and Welling, 2016; Veli ˇ ckovi ´ c et al., 2018; Schlichtkrull et al., 2018), and attention neural networks (Jaderberg et al., 2015; Wang et al., 2017), have been widely applied for various artificial intelligence (AI) tasks in recent years.”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, methodology): “Self-attention in CNNs Self-attention mechanism has been widely applied to CNNs in vision task [38, 57, 19, 47, 20, 39, 1, 6, 18, 31, 42, 13, 50, 48].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Early work studied hybrid models consisting of both convolutions and attention modules [94, 100, 40, 111].”
Selective Kernel Networks (2019, background): “[44] propose a trunk-and-mask attention between intermediate stages of a CNN.”
An Empirical Study of Spatial Attention Mechanisms in Deep Networks (2019, background): “It is worth mentioning a dual form of spatial attention, called channel-wise feature attention [37, 44, 21, 13].”
Image Super-Resolution Using Very Deep Residual Channel Attention Networks (2018, background): “[26] proposed residual attention network for image classification with a trunk-and-mask attention mechanism.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Several works have included attention mechanisms in neural network architectures designed for vision [35, 36, 37, 38].”
Attention-based Deep Multiple Instance Learning (2018, neutral): “Since attention serves as a gradient update ﬁlter during backpropagation (Wang et al., 2017), instances with higher weights will contribute more to learning the encoder network of instances.”