Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks

By Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, Shimin Hu. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021

709 citations recorded by the source corpus.

Attention mechanisms, especially self-attention, have played an increasingly important role in deep feature representation for visual tasks. Self-attention updates the feature at each position by computing a weighted sum of features using pair-wise affinities across all positions to capture the long-range dependency within a single sample. However, self-attention has quadratic complexity and ignores potential correlation between different samples. This article proposes a novel attention mechanism which we call external attention, based on two external, small, learnable, shared memories, which can be implemented easily by simply using two cascaded linear layers and two normalization layers; it conveniently replaces self-attention in existing popular architectures. External attention has linear complexity and implicitly considers the correlations between all data samples. We further incorporate the multi-head mechanism into external attention to provide an all-MLP architecture, external attention MLP (EAMLP), for image classification. Extensive experiments on image classification, object detection, semantic segmentation, instance segmentation, image generation, and point cloud analysis reveal that our method provides results comparable or superior to the self-attention mechanism and some of its variants, with much lower computational and memory costs.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fc92009ab34045f9e6d490684c7761f768e88c54

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Ming-Ming Cheng
3. Yunhe Wang
4. Qibin Hou
5. Kai Han
6. Jing Zhang
7. Jianyuan Guo
8. Chunjing Xu

Citations in the corpus

• MALUNet: A Multi-Attention and Light-weight UNet for Skin Lesion Segmentation (2022, methodology): “The modules in recent works are often improved based on the SA module or its variants, while SA is not expert in modeling the relationship between samples [16].”
• A Survey on Vision Transformer (2020, background): “, positional encoding [44], [242], normalization strategy [220], shortcut connection [215] and removing attention [217], [158], [79], [218].”
• Attention mechanisms in computer vision: A survey (2021, background): “EANet [75] proposes that self-attention should only consider correlation in a single sample and should ignore potential relationships between different samples.”
• Mixed Transformer U-Net for Medical Image Segmentation (2021, methodology): “To tackle these problems, we redesign SA for better lo-cal perception with lower computational cost, then integrating it with External Attention (EA) [12] to manage inter-and intra-correlations simultaneously.”
• UIU-Net: U-Net in U-Net for Infrared Small Object Detection (2022, background): “[39], [40], [41], [42], [43] have also been proposed and widely used in many fields, including image classification [40], [41], semantic segmentation [40], object detection [42], and more, with good results.”
• SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, background): “…types of decoders for different goals, including achieving multi-scale receptive fields [94, 7, 78], collecting multi-scale semantics [64, 80, 8], enlarging receptive field [5, 4, 62], strengthening edge features [95, 2, 16, 42, 90], and capturing global context [19, 34, 89, 40, 23, 26, 91].”
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “Following works along this direction are to simplify the model structures with fewer intrinsic IBs and directly learn them from large scale data [42, 63, 64, 18, 15], which have achieved promising results and been studied actively.”
• Continual Learning with Lifelong Vision Transformer (2022, methodology): “We use a external key K W [30] to replace input-depended self-key, and explicitly add a learnable attention bias B to the attention maps.”
• Visual attention network (2022, methodology): “However, due to high computational requirements and low efficiency, the capability of MLPs was been limited in a long time.”