Global Self-Attention Networks for Image Recognition

By Zhuoran Shen, Irwan Bello, Raviteja Vemulapalli, Xuhui Jia, Ching-Hui Chen. 2020

33 citations recorded by the source corpus.

Recently, a series of works in computer vision have shown promising results on various image and video understanding tasks using self-attention. However, due to the quadratic computational and memory complexities of self-attention, these works either apply attention only to low-resolution feature maps in later stages of a deep network or restrict the receptive field of attention in each layer to a small local region. To overcome these limitations, this work introduces a new global self-attention module, referred to as the GSA module, which is efficient enough to serve as the backbone component of a deep network. This module consists of two parallel layers: a content attention layer that attends to pixels based only on their content and a positional attention layer that attends to pixels based on their spatial locations. The output of this module is the sum of the outputs of the two layers. Based on the proposed GSA module, we introduce new standalone global attention-based deep networks that use GSA modules instead of convolutions to model pixel interactions. Due to the global extent of the proposed GSA module, a GSA network has the ability to model long-range pixel interactions throughout the network. Our experimental results show that GSA networks outperform the corresponding convolution-based networks significantly on the CIFAR-100 and ImageNet datasets while using less parameters and computations. The proposed GSA networks also outperform various existing attention-based networks on the ImageNet dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Training data-efficient image transformers & distillation through attention (2020, background): “More recently several researchers have proposed hybrid architecture transplanting transformer ingredients to convnets to solve vision tasks [6, 40].”
ViViT: A Video Vision Transformer (2021, methodology): “Although previous works attempted to replace convolu-tions in vision architectures [49, 53, 55], it is only very recently that Dosovitisky et al .”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “…attention patterns (Dai et al., 2019; Parmar et al., 2018; Ramachandran et al., 2019) • axial attention patterns (Ho et al., 2019; Wang et al., 2020a; Shen et al., 2020) • static sparse attention patterns (Child et al.; Beltagy et al., 2020) • dynamic sparse attention patterns (Kitaev et al.,…”
Bottleneck Transformers for Visual Recognition (2021, background): “Our paper does not touch upon comparisons to backbones that employ group convolutions [20, 10], concurrent work on channel (or linear) attention in c4 [1], axial attention [18, 13] and local attention [11], but we remark that changes prior to the c5 blockgroup can benefit BoTNets as well.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Therefore, we additionally report raw timings on reference hardware, like recent papers [2, 58].”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, methodology): “There have also been efforts in going beyond standard ConvNets for image classification, by adapting self-attention [56] to the visual domain [2, 40, 20, 47, 8, 1].”
Generative Adversarial Transformers (2021, background): “Few other works proposed sparse, discrete or approximated variations of self-attention, either within the adversarial or autoregressive contexts, but they still fall short of reducing memory footprint and computation costs to a sufficient degree [11, 24, 37, 39, 63].”