On the Integration of Self-Attention and Convolution

By Xuran Pan, Chunjiang Ge, Rui Lu, Shiji Song, Guanfu Chen, Zeyi Huang, Gao Huang. Computer Vision and Pattern Recognition, 2021

490 citations recorded by the source corpus.

Convolution and self-attention are two powerful techniques for representation learning, and they are usually considered as two peer approaches that are distinct from each other. In this paper, we show that there exists a strong underlying relation between them, in the sense that the bulk of computations of these two paradigms are in fact done with the same operation. Specifically, we first show that a traditional convolution with kernel size k × k can be decomposed into k2 individual 1 × 1 convolutions, followed by shift and summation operations. Then, we interpret the projections of queries, keys, and values in self-attention module as multiple 1 × 1 convolutions, followed by the computation of attention weights and aggregation of the values. Therefore, the first stage of both two modules comprises the similar operation. More importantly, the first stage contributes a dominant computation complexity (square of the channel size) comparing to the second stage. This observation naturally leads to an elegant integration of these two seemingly distinct paradigms, i.e., a mixed model that enjoys the benefit of both self-Attention and Convolution (ACmix), while having minimum compu-tational overhead compared to the pure convolution or self-attention counterpart. Extensive experiments show that our model achieves consistently improved results over com-petitive baselines on image recognition and downstream tasks. Code and pre-trained models will be released at https://github.com/LeapLabTHU/ACmix and https://gitee.com/mindspore/models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a0023d03985f94dddef12f762bda45948f144460

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Shiji Song
2. Ser-Nam Lim
3. Gao Huang
4. Jie Zhou
5. Yizeng Han
6. Jiwen Lu
7. Xuran Pan
8. Yongming Rao

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Jie Zhou
3. Jiwen Lu
4. Gao Huang
5. Yongming Rao
6. Ser-Nam Lim
7. Yizeng Han
8. Luc Van Gool

Citations in the corpus

• HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, background): “Recently, several efforts have been made to integrate the two types of blocks into a unified model with a sequential [12, 28, 64, 59] or parallel [43, 11] design.”
• Vision Transformer with Deformable Attention (2022, background): “Since the introduction of ViT [12], improvements [6, 11, 26, 28, 36, 43, 49] have focused on learning multi-scale features for dense prediction tasks and efficient attention mechanisms.”
• FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “With the advent of Vision Transformer [11, 39], self-attention techniques have shown great potential in a variety of vision tasks including image classification [41, 43, 30, 46], semantic segmentation [6, 49], object detection [4, 61, 22], and multi-modal tasks [35, 31].”