MLP-Mixer: An all-MLP Architecture for Vision

By Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic and 1 more. 2021

3,705 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e."mixing"the per-location features), and one with MLPs applied across patches (i.e."mixing"spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (2021, neutral): “, 2019), alternative non Transformer-based architectures (Tolstikhin et al., 2021; Tay et al., 2021a; 2020a; Lee-Thorp et al., 2021).”
A Review of Sparse Expert Models in Deep Learning (2022, methodology): “(2021) introduces a sparse MoE MLP model for image classification based on the MLP-Mixer architecture (Tolstikhin et al., 2021).”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, background): “Other choices are convolutional neural network (CNN) or all-MLP [44] structures, but it is unclear how to effectively apply such networks to all the three roles.”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “We verify LiT across three image-text datasets, with Vision Transformer [20], ResNet [32], and MLP-Mixer [60] architectures.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “The shifted window approach also proves beneficial for all-MLP architectures [56].”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “, block designs [50,51], self-supervised learning [2,23], and scaling [55].”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “Ever since [18] first showed good results when pre-training BiT on ImageNet-21k, more architecture works have mentioned using it for select few experiments [10, 33, 32, 7], with [26] arguing more directly for the use of ImageNet-21k.”
Efficient Transformers: A Survey (2020, background): “Amongst the many alternatives considered, a prominent line of emerging research belongs to the family of MLP Mixers (Tolstikhin et al., 2021).”
Novel Class Discovery without Forgetting (2022, background): “Over the last decade, deep learning algorithms have achieved remarkable performances on multiple computer vision tasks [15,37,50,7,45], even outperforming humans on many of them.”
Transformers in Vision: A Survey (2021, methodology): “For example, MLPMixer [196] directly operates on input image patches and separately performs token and channel mixing without any dedicated classiication tokens as in ViT.”
Synthesizer: Rethinking Self-Attention for Transformer Models (2020, background): “The key difference here is that (1) we use a softmax normalization on the kernel (weights) and (2) Random Synthesizers are a form of multi-headed MLP-Mixers.”
ImageNet-21K Pretraining for the Masses (2021, methodology): “Our proposed pretraining scheme also outperforms previous ImageNet-21K pretraining schemes that were used to trained MLP-based models like Vision-Transformer (ViT) [14] and Mixer [53].”
ConViT: improving vision transformers with soft convolutional inductive biases (2021, neutral): “More recently, the data-eﬃcient vision transformer (DeiT) (Touvron et al 2020) was able to reach similar performances without any pre-training on supplementary data, instead relying on knowledge distillation (Hinton et al 2015) from a convolutional teacher.”
FNet: Mixing Tokens with Fourier Transforms (2021, methodology): “Our Linear model is similar to the MLP-Mixer (Tolstikhin et al., 2021) (for vision) and also the Random Syn-thesizer (Tay et al., 2020a), but simpliﬁes the latter model further by removing the multiple heads and softmax projections, resulting in just two matrix multiplications in the mixing…”
PaLI: A Jointly-Scaled Multilingual Language-Image Model (2022, background): “, 2021), and other models (Tolstikhin et al., 2021; Riquelme et al., 2021) have seen similar benefits from scale (Zhai et al.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Recently data-independent analogues of self-attention have also been found to be an effective alternative to convolutional and self-attention layers for vision tasks [20, 46, 62, 66].”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, background): “Beyond LRDs, these benchmarks point to a recent eﬀort of the ML community to solve vision problems with reduced domain knowledge, in the spirit of models such as Vision Transformers [12] and MLP-Mixer [41]”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Before the grouping block, we project the 64 group tokens into 8 tokens using an MLP-Mixer layer [71] and output 8 segment tokens.”
Mobile-Former: Bridging MobileNet and Transformer (2021, methodology): “position-mixing MLP: Table 7 shows the results of replacing multi-head attention (MHA) with token/position mixing MLP [33] in both Former and bridge (Mobile→Former and Mobile←Former).”
MaxViT: Multi-Axis Vision Transformer (2022, methodology): “Moreover, we can also adopt the proposed multi-axis approach on the MLP-Mixer [78] or gMLP [52] model to improve the scalability and efficiency, which we leave to future research.”