Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

By Luke Melas-Kyriazi. 2021

119 citations recorded by the source corpus.

The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature dimensions in an alternating fashion. In experiments on ImageNet, this architecture performs surprisingly well: a ViT/DeiT-base-sized model obtains 74.9\% top-1 accuracy, compared to 77.9\% and 79.9\% for ViT and DeiT respectively. These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought. We hope these results prompt the community to spend more time trying to understand why our current models are as effective as they are.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, neutral): “The architectural improvements are also pertinent in view of recent results [7, 142, 195, 196] that demonstrate fairly simple architectures can lead to competitive performance on a range of vision problems.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Recently data-independent analogues of self-attention have also been found to be an effective alternative to convolutional and self-attention layers for vision tasks [20, 46, 62, 66].”
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “Following works along this direction are to simplify the model structures with fewer intrinsic IBs and directly learn them from large scale data [42, 63, 64, 18, 15], which have achieved promising results and been studied actively.”
A Survey on Vision Transformer (2020, background): “, positional encoding [44], [242], normalization strategy [220], shortcut connection [215] and removing attention [217], [158], [79], [218].”
Pay Attention to MLPs (2021, background): “Moreover, the accuracy-parameter/FLOPs tradeoff of gMLPs surpasses all concurrently proposed MLP-like architectures [20, 21, 22], which we attribute to the effectiveness of our Spatial Gating Unit (see Table 3 in the next section for an ablation).”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, methodology): “Spatial MLPs Recent works [12, 24, 31, 48, 49] on MLP-only architectures in NLP and CV replace the self-attention layer in Transformers with MLPs across tokens or spatial locations.”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, background): “In computer vision, there has recently been a surge of interest in end-to-end Transformers [1, 2, 5, 20, 22, 38, 55] and MLPs [37, 40, 53, 54], prompting the efforts to replace hand-wired features or inductive biases with general-purpose neural architectures powered by datadriven training.”