On the Relationship between Self-Attention and Convolutional Layers

By Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi. 2019

645 citations recorded by the source corpus.

Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “Most related to ours is the model of Cordonnier et al. (2020), which extracts patches of size 2 × 2 from the input image and applies full self-attention on top.”
End-to-End Object Detection with Transformers (2020, methodology): “It starts by computing so-called query, key and value embeddings after adding the query and key positional encodings [7]: where T (cid:48) is the concatenation of T 1 , T 2 , T 3 .”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “[18, 13, 21, 17, 15] on the other hand restrict receptive field of self-attention and use it in a convolutional manner.”
Training data-efficient image transformers & distillation through attention (2020, background): “(Cordonnier et al., 2020) study the link between SA and convolution.”
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth (2021, background): “There have been other recent advances in understanding the theory behind transformers: [PMB19, DGV + 19] proved Turing universality, [CLJ20b] provided necessary and sufficient conditions for attention to simulate convolution.”
Scaling Vision Transformers (2021, background): “Smaller Vision Transformers Early work on Transformers for vision focused on small networks for CIFAR10 [10].”
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “Works that use only self-attention blocks tend to apply them at an individual pixel level [12, 39, 43], often requiring tricks to ensure computational tractability, including restricting the scope of self-attention to a local neighbourhood [43], adding global self-attention on heavily downsized vers”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020, background): “More recently, Cordonnier et al. (2020) provided theoretical proofs and empirical evidence that a multi-head selfattention with sufficient number of heads can express any convolutional layer.”
Group Equivariant Stand-Alone Self-Attention For Vision (2020, background): “A.1), yet self-attention layers are able to express any convolutional layer (Cordonnier et al., 2020).”
Transformers in Vision: A Survey (2021, background): “Cordonnier et al.[45] further studied the relationships between self-attention and convolution operations.”
MLP-Mixer: An all-MLP Architecture for Vision (2021, background): “Their value for image recognition was demonstrated later, albeit in combination with a convolution-like locality bias [37], or on low-resolution images [10].”
Synthesizer: Rethinking Self-Attention for Transformer Models (2020, background): “Meanwhile, in the image domain, (Cordonnier et al., 2019) shows connection of Transformers with CNNs.”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “This broadens the family of functions they can represent (Cordonnier et al., 2020; Zhao et al., 2020), and renders them better suited to modern big-data regimes where there is less need for strong inductive priors.”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, background): “Recent work [8] has shown that self-attention with sufficient number of heads and the right geometric biases can simulate convolutions, suggesting a deeper relationship between self-attention and convolutions.”
ConViT: improving vision transformers with soft convolutional inductive biases (2021, methodology): “Indeed, Cordonnier, Loukas and Jaggi [32] showed that a SA layer with Nh heads can express a convolution of kernel size √ Nh, if each head focuses on one of the pixels in the kernel patch.”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Due to the quadratic computational complexity and number of parameters involved in attention mechanisms, most authors [23, 24] initially considered images of small sizes like in CIFAR or Imagenet64 [25].”
Involution: Inverting the Inherence of Convolution for Visual Recognition (2021, background): “Still, limited emphasis has been laid on delving deep into the learning dynamics of this functional form compared to convolution [7].”