Pay Attention to MLPs

By Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le. Neural Information Processing Systems, 2021

883 citations recorded by the source corpus.

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e3a3e85c5a32af29e13b3561f6cf070de70651de

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Sparse-MLP: A Fully-MLP Architecture with Conditional Computation (2021, methodology): “Fully MLP-based models recently achieved promising results in vision (Tolstikhin et al. 2021; Liu et al. 2021a; LeeThorp et al. 2021; Hou et al. 2021).”
Efficient Transformers: A Survey (2020, methodology): “Diﬀerent mixing operations have been proposed, such as the G-MLP (Liu et al., 2021a), FNet (Lee-Thorp et al., 2021).”
Are Transformers More Robust Than CNNs? (2021, background): “We hope this work can serve as a useful benchmark for future explorations on robustness, using different network architectures, like CNNs, Transformers, and beyond [24, 40].”
MaxViT: Multi-Axis Vision Transformer (2022, methodology): “Moreover, we can also adopt the proposed multi-axis approach on the MLP-Mixer [78] or gMLP [52] model to improve the scalability and efficiency, which we leave to future research.”
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT (2022, methodology): “Orthogonal to MoE, efficient mixing models (Tolstikhin et al., 2021; Liu et al., 2021; Lee-Thorp et al., 2021) replace attention in Transformer-like models with simpler linear transformations or MLP blocks that "mix" input representations.”
Transformer Quality in Linear Time (2022, background): “Consistent with the ﬁndings in Liu et al. (2021), the presence of gating allows the use of a much simpler/weaker attention mechanism than MHSA without quality loss: where Z is a shared representation ( s d ) 4 , Q and K are two cheap transformations that apply per-dim scalars and offsets to Z…”
Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition (2021, background): “Very recently, there are also some work [1], [2], [32], [33] targeting at developing pure MLP-like models for ImageNet classification.”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, methodology): “Spatial MLPs Recent works [12, 24, 31, 48, 49] on MLP-only architectures in NLP and CV replace the self-attention layer in Transformers with MLPs across tokens or spatial locations.”
Red Alarm for Pre-trained Models: Universal Vulnerabilities by Neuron-Level Backdoor Attacks (2021, background): “Large-scale pre-training has achieved great success in NLP and CV, giving birth to many well-known PTMs [1, 11, 13, 14, 15, 12, 16, 17].”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, background): “More recent works ﬁnd that the self-attention in ViT is not vital for performance, resulting in several architectures exclusively based on MLPs (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a; Melas-Kyriazi, 2021).”
DaViT: Dual Attention Vision Transformers (2022, background): “Most recently, MLP-based models [53,54,39,74,8,35,27] draw attention by using multi-layer perceptrons (MLPs) directly over patch features.”
MetaFormer is Actually What You Need for Vision (2021, background): “Even compared with more improved ViT and MLPlike variants [34, 55], PoolFormer still shows better performance.”
Focal Modulation Networks (2022, methodology): “We see that FocalNets outperform the conventional CNNs ( e.g. , ResNet [27] and the augmented version [73]), MLP architectures such as MLP-Mixer [61] and gMLP [43], and Transformer architectures DeiT [63] and PVT [69].”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, background): “Recent works have been focused on improving the architectures of attention models, including sparse attention [33, 34, 35, 36, 37, 38, 39], enlarged receptive fields [40, 41], pyramidal designs [42, 43, 44], efficient alternatives [45, 46, 47], etc.”