An Empirical Study of Spatial Attention Mechanisms in Deep Networks

By Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, Jifeng Dai. 2019

543 citations recorded by the source corpus.

Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the query and key content comparison in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. A proper combination of deformable convolution with key content only saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “On semantic segmentation, self-attention is developed as a context aggregation module that captures multi-scale context [42,27,102,99].”
Dynamic Neural Networks: A Survey (2021, background): “Readers who are interested in this topic may refer to review studies [139], [140], [141].”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, methodology): “On the other hand, as discussed in Zhu et al. (2019a), there are variants of convolution, such as deformable convolution (Dai et al., 2017; Zhu et al., 2019b) and dynamic convolution (Wu et al., 2019), that also can be viewed as self-attention mechanisms.”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, background): “(2) Regular convolutions can be deemed as special instantiations of spatial attention mechanisms [16].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
MMDetection: Open MMLab Detection Toolbox and Benchmark (2019, background): “• Generalized Attention [41]: a generalized attention formulation, proposed in 2019.”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, background): “Previous works [21, 22, 50] have extensively discussed the differences between CNNs and ViTs.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “More works [21,40,44,100,121] explained the superiority of ViTs from different point of views.”
Attention mechanisms in computer vision: A survey (2021, methodology): “[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90”
FcaNet: Frequency Channel Attention Networks (2020, methodology): “Inspired by these works, a series of works like BAM [28], DAN [9], CBAM [39], scSE [32], and CoordAtten-tion [19] are proposed to fuse spatial attention [45] and channel attention.”
Disentangled Non-Local Neural Networks (2020, background): “Our work is also related to several approaches that analyze the non-local/self-attention mechanism in depth, including the performance of individual terms [21,32,46] on various tasks.”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, background): “Both classes of methods have boosted various vision tasks, such as image classification [6, 17, 27, 38, 55, 62, 72, 83], object detection [10, 37, 72,76,85,96], semantic segmentation [15,28,40,93,95,97], video recognition [17, 45, 85], image generation [36, 69], and panoptic segmentation [83].”
Latent Video Transformer (2020, unknown): “ams (M) inference time (sec) query-key + relative distance 1:53 125:76 2:90 49:87 35 key-only + relative distance 1:57 130:27 4:26 41:50 32 relative distance only 1:58 141:62 4:34 33:09 30 Similar to [52] we ﬁnd that query-key term inside self-attention module does not play a crucial role in the suc”
Involution: Inverting the Inherence of Convolution for Visual Recognition (2021, background): “Its success has also been translated to a plethora of vision tasks, including image recognition [2, 17, 31, 53], image generation [29, 51], video understanding [36, 44], object detection [3, 16, 54], and semantic segmentation [12, 18, 42].”
k-means Mask Transformer (2022, background): “Both approaches have boosted various vision tasks, such as image classiﬁcation [19,7,81,44,64,94,32,70,107,103], image generation [78,42], object detection [97,84,81,43,10,116], video recognition [97,19,3,33], semantic segmentation [17,110,46,35,117,115,111,100,11], and panoptic segmentation [94].”
CenterMask: Real-Time Anchor-Free Instance Segmentation (2019, background): “Recently, attention methods [13, 34, 40, 28] have been widely applied to object detections because it helps to focus on important features, but also suppress unnecessary ones.”