Scaling Local Self-Attention for Parameter Efficient Visual Backbones

By Ashish Vaswani, Prajit Ramachandran, A. Srinivas, Niki Parmar, Blake A. Hechtman, Jonathon Shlens. 2021

476 citations recorded by the source corpus.

Self-attention has the promise of improving computer vision systems due to parameter-independent scaling of receptive fields and content-dependent interactions, in contrast to parameter-dependent scaling and content-independent interactions of convolutions. Self-attention models have recently been shown to have encouraging improvements on accuracy-parameter trade-offs compared to baseline convolutional models such as ResNet-50. In this work, we develop self-attention models that can outperform not just the canonical baseline models, but even the high-performing convolutional models. We propose two extensions to self-attention that, in conjunction with a more efficient implementation of self-attention, improve the speed, memory usage, and accuracy of these models. We leverage these improvements to develop a new self-attention model family, HaloNets, which reach state-of-the-art accuracies on the parameter-limited setting of the ImageNet classification benchmark. In preliminary transfer learning experiments, we find that HaloNet models outperform much larger models and have better inference performance. On harder tasks such as object detection and instance segmentation, our simple local self-attention and convolutional hybrids show improvements over very strong baselines. These results mark another step in demonstrating the efficacy of self-attention models on settings traditionally dominated by convolutions. 1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Scaling Vision Transformers (2021, methodology): “Hybrids of CNNs and Transformers typically allow smaller models to perform well, such as the ViT-CNN hybrid in [15], BoTNet [35], and HaloNet [43].”
EDTER: Edge Detection with Transformer (2022, background): “This architecture brings direct inspiration to other computer vision tasks [32, 38, 55, 68, 72].”
The Efficiency Misnomer (2021, background): “Another example for how implementation details can affect the efﬁciency is presented in (Vaswani et al., 2021), where the author discusses how carefully designed implementation of the local neighborhood gathering function for local attention might reduce memory usage while avoiding unnecessary…”
Online Continual Learning with Contrastive Vision Transformer (2022, background): “Since then, a lot of effort has been dedicated to improving Vision Transformers’ data efficiency and model efficiency [92,32,91,40], where an effective direction is to strategically integrate properties of convolution into the Transformer architecture [73,91,48,84,26,94,23,17].”
Transformers in Vision: A Survey (2021, methodology): “Numerous methods have been proposed that make special design choices to perform self-attention more “efficiently,” for instance employing pooling/downsampling in self-attention [61, 208, 220], local window-based attention [137, 200],”
MLP-Mixer: An all-MLP Architecture for Vision (2021, methodology): “HaloNets are attention-based models that use a ResNet-like structure with local selfattention layers instead of 3×3 convolutions [51].”
Palette: Image-to-Image Diffusion Models (2021, background): “Local Self-Attention: Local self-attention layers [Vaswani et al., 2021] at 32×32, 16×16 and 8×8 resolutions, at which feature maps are divided into 4 non-overlapping query blocks.”
Bottleneck Transformers for Visual Recognition (2021, neutral): “Our implementation of drop-connect and the balance between RA and WD is similar to HaloNets [16].”
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding (2021, methodology): “Swin Transformer [26] and HanoNet [44] utilizes similar local attention mechanism as our Vision Longformer.”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Recent works alternatively exploited either a coarse-grained global self-attention [63, 67] or a fine-grained local self-attention [44, 80, 59] to reduce the computational burden.”
Dilated Neighborhood Attention Transformer (2022, methodology): “Works succeeding it therefore switched to alternative methods that could run more efficiently, such as blocked self attention in HaloNet [38], and Window Self Attention in Swin [26].”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, methodology): “A2), as used by Halonet (Vaswani et al. 2021), performs poorly on data efficient benchmarks.”
Long-Short Transformer: Efficient Transformers for Language and Vision (2021, background): “HaloNet [41] applies local attention on blocked images and only has quadratic complexity with respect to the size of the block.”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “To overcome the quadratic complexity of original self-attention, self-attention is performed on local windows with the halo operation or window shift [40, 55] to help cross-window interaction, and get promising re-sults.”
Visformer: The Vision-friendly Transformer (2021, methodology): “Besides utilizing convolution in early layers [30], BotNet [29] designs bottleneck cells for self-attention.”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, background): “HaloNets [39] develops a new self-attention model family that are simple local self-attention and convolutional hybrids, which outperform both CNN and vanilla ViT on a range of downstream vision tasks.”
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows (2021, methodology): “Vision Transformers.”
DaViT: Dual Attention Vision Transformers (2022, methodology): “To further reduce the computational cost, local attention [37,75,52] that limits attention in a spatially local window, and squeezed projection [58,55] that performs attention on downsampled tokens, were proposed.”