Visual attention network

By Meng-Hao Guo, Chengrou Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shiyong Hu. Computational Visual Media, 2022

1,051 citations recorded by the source corpus.

While originally designed for natural language processing tasks, the self-attention mechanism has recently taken various computer vision areas by storm. However, the 2D nature of images brings three challenges for applying self-attention in computer vision: (1) treating images as 1D sequences neglects their 2D structures; (2) the quadratic complexity is too expensive for high-resolution images; (3) it only captures spatial adaptability but ignores channel adaptability. In this paper, we propose a novel linear attention named large kernel attention (LKA) to enable self-adaptive and long-range correlations in self-attention while avoiding its shortcomings. Furthermore, we present a neural network based on LKA, namely Visual Attention Network (VAN). While extremely simple, VAN achieves comparable results with similar size convolutional neural networks (CNNs) and vision transformers (ViTs) in various tasks, including image classification, object detection, semantic segmentation, panoptic segmentation, pose estimation, etc. For example, VAN-B6 achieves 87.8% accuracy on ImageNet benchmark, and sets new state-of-the-art performance (58.2 PQ) for panoptic segmentation. Besides, VAN-B2 surpasses Swin-T 4 mIoU (50.1 vs. 46.1) for semantic segmentation on ADE20K benchmark, 2.6 AP (48.8 vs. 46.2) for object detection on COCO dataset. It provides a novel method and a simple yet strong baseline for the community. The code is available at https://github.com/Visual-Attention-Network.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ba637c4f1a170f1e2dadeadb71a63cf2b9a46de2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Rewrite the Stars (2024, positive): “Meanwhile, both VAN [18] and Monarch Mixer [16] attribute the effectiveness to convolutional attention.”
Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN (2023, methodology): “Such longrange dependency in CNN can be modeled by using larger receptive fields with attention mechanisms [34].”
TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution (2024, background): “To reduce the complexity of large kernel CNN, Guo et al. [50] proposed a large kernel convolutional decomposition strategy (LKA) without sacrificing the respective field.”
MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 58, 81], MLP-based [67, 68, 27, 6, 41], convolution-based [15, 19, 80, 56, 78], hybrid [62, 12, 37, 71] or other types of [65, 20] token mixers.”
Poly Kernel Inception Network for Remote Sensing Detection (2024, methodology): “This is achieved by incorporating large-kernel convolutions [12, 18, 38, 43] and dilated convolutions into the backbone network.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, methodology): “As a comparison, we follow VAN [24] and replace the multiple branches in our MSCA with a single convolution with a large kernel.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “VAN [19] and FocalNet [61] use gated convolutions to perform input-adaptive attention and adopts large-kernel dilated convolutions and multiple successive 3×3 convolutions respectively to produce the weights.”
Fast Vision Transformers with HiLo Attention (2022, background): “Second, some efficient attention mechanisms in ViTs have low theoretical complexity guarantee but are actually slow on GPUs due to particular operations that are not hardware-friendly or cannot be parallelized, such as the multi-scale window partition [48], recursion [35] and dilated window [15].”
InceptionNeXt: When Inception Meets ConvNeXt (2023, positive): “ConvNeXt is not alone: More and more works have shown similar observations [14, 18, 24, 38, 51, 68, 76, 78], like RepLKNet [14] and SLaK [38].”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “…of decomposed large kernels are weighted by their corresponding spatial selection masks and then fused by a convolution layer F ( · ) to obtain the attention feature S : The final output of the LSK module is the element-wise product between the input feature X and S , similarly in [19,20,28]: Fig.”
Attention mechanisms in computer vision: A survey (2021, background): “[148, 123, 51]) and use it to select important features.”
Focal Modulation Networks (2022, methodology): “We notice three concurrent works, including ConvNeXT [42], RepLKNet [14] and Visual Attention Networks (VAN) [20].”