Attention mechanisms in computer vision: A survey

By Meng-Hao Guo, Tianhan Xu, Jiangjiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph Robert Martin, Ming-Ming Cheng, Shimin Hu. Computational Visual Media, 2021

2,419 citations recorded by the source corpus.

Humans can naturally and effectively find salient regions in complex scenes. Motivated by this observation, attention mechanisms were introduced into computer vision with the aim of imitating this aspect of the human visual system. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. Attention mechanisms have achieved great success in many visual tasks, including image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multimodal tasks, and self-supervised learning. In this survey, we provide a comprehensive review of various attention mechanisms in computer vision and categorize them according to approach, such as channel attention, spatial attention, temporal attention, and branch attention; a related repository https://github.com/MenghaoGuo/Awesome-Vision-Attentions is dedicated to collecting related work. We also suggest future directions for attention mechanism research.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/45f686be3b96302ede327645227134e1c304dbab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Convolutional Neural Networks: A Survey (2023, positive): “Attention mechanisms are a powerful tool for improving the performance of CNNs [178–182].”
Masked Diffusion Transformer is a Strong Image Synthesizer (2023, methodology): “Several design improvements, e.g ., adding more attention heads [18], BigGAN [4] residual block, and adaptive group normalization, are proposed in [11] to further enhance the generation ability of the UNet.”
Transformers in Medical Imaging: A Survey (2022, background): “Further, we find it important to clarify that several alternate attention approaches [90]–[93] have been explored in the literature based on convolutional architectures.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, background): “Additionally, there is a recently published review on attention mechanisms in computer vision [45], albeit with a limited scope, where it primarily concentrates on limited and basic attention mechanisms such as spatial, temporal, channel, and branch attention methods.”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, background): “The attention mechanism [21] is a simple but effective way to enhance neural representations for various tasks.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, background): “Generally speaking, it can be divided into two categories in semantic segmentation [25], including channel attention and spatial attention.”
Visual attention network (2022, background): “For vision tasks, different channels often represent different objects [43], [65].”
Attention Spiking Neural Networks (2022, background): “Please refer to [44] for a comprehensive review of attention in CV.”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “Many attention models are designed based on this idea, such as the channel attention, spatial attention, temporal attention, branch attention [129].”
Spike-driven Transformer (2023, background): “14 actually converts the soft vanilla self-attention to hard self-attention, where the attention scores in soft and hard attention are continuous-and binary-valued, respectively [75].”