Neighborhood Attention Transformer

By Ali Hassani, Steven Walton, Jiacheng Li, Shengjia Li, Humphrey Shi. 2022

500 citations recorded by the source corpus.

We present Neighborhood Attention (NA), the first efficient and scalable sliding window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a linear time and space complexity compared to the quadratic complexity of SA. The sliding window pattern allows NA's receptive field to grow without needing extra pixel shifts, and preserves translational equivariance, unlike Swin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood Attention Extension), a Python package with efficient C++ and CUDA kernels, which allows NA to run up to 40% faster than Swin's WSA while using up to 25% less memory. We further present Neighborhood Attention Transformer (NAT), a new hierarchical transformer design based on NA that boosts image classification and downstream vision performance. Experimental results on NAT are competitive; NAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MSCOCO and 48.4% mIoU on ADE20K, which is 1.9% ImageNet accuracy, 1.0% COCO mAP, and 2.6% ADE20K mIoU improvement over a Swin model with similar size. To support more research based on sliding window attention, we open source our project and release our checkpoints.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Yizeng Han
3. Gao Huang
4. Xuran Pan
5. Feng Zhu
6. Dongchen Han
7. Kun-Li Channing Lin
8. Zhihao Duan

In Electrical and Electronic Engineering

1. Shiji Song
2. Gao Huang
3. Xuran Pan

In Model Architectures

1. Shiji Song
2. Gao Huang
3. Kun-Li Channing Lin
4. Feng Zhu
5. Yipeng Gao
6. Yaowei Wang
7. Yuyao Tang
8. Wei-Shi Zheng

Citations in the corpus

• DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “[35], [41], [51], [53], [70], [93], [114] is beneficial for dense prediction tasks such as object detection, instance and semantic segmentation, recent works [21], [23], [32], [38], [42], [54], [58], [66], [77], [78], [95], [104] introduce multi-scale modeling capability by using a pyramid structure”
• MambaOut: Do We Really Need Mamba for Vision?* (2024, unknown): “Numerous studies [78] have explored various strategies to mitigate this issue, including low-rank approaches [84], kernelization [14, 45], token mixing range limitation [7, 33, 57, 105], and history memory compression [68].”
• VMamba: Visual State Space Model (2024, unknown): “2% mIoU on instance segmentation, respectively.”
• QARV: Quantization-Aware ResNet VAE for Lossy Image Compression (2023, methodology): “We adopt a recent Vision Transformer architecture, the Neighbour-hood Attention Transformer (NAT) [60], to use in QARV.”
• FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “Previous works have sought to mitigate this challenge by confining the global receptive field to a smaller region, such as designing sparse global attention patterns [41, 46] or applying smaller attention windows [24, 17].”