ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training

By Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek and 1 more. 2021

901 citations recorded by the source corpus.

We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “, block designs [50,51], self-supervised learning [2,23], and scaling [55].”
Transformers in Vision: A Survey (2021, methodology): “The architectural improvements are also pertinent in view of recent results [7, 142, 195, 196] that demonstrate fairly simple architectures can lead to competitive performance on a range of vision problems.”
XCiT: Cross-Covariance Image Transformers (2021, background): “Recently data-independent analogues of self-attention have also been found to be an effective alternative to convolutional and self-attention layers for vision tasks [21, 46, 64, 68].”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “Later research further improves ViTs by incorporating hierarchical de-sign [74, 71], injecting locality with the aid of convolu-tions [25, 17, 27, 66], or exploring different types of to-ken mixing such as local attention [50, 19], spatial MLP mixer [69, 68], and non-parameterized pool mixer [83].”
Transformers in Medical Image Analysis: A Review (2022, background): “Simultaneously with or following MLP-Mixer, many other MLP-based models were proposed, e.g., gMLP [33], ResMLP [34], ASMLP [35], CycleMLP [36], etc. MLP-Mixer not only inspired further exploration of MLP-based model, but also raised stimulated further developments of neural architectures in CV.”
When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations (2021, background): “More recent works ﬁnd that the self-attention in ViT is not vital for performance, resulting in several architectures exclusively based on MLPs (Tolstikhin et al., 2021; Touvron et al., 2021a; Liu et al., 2021a; Melas-Kyriazi, 2021).”
TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting (2023, methodology): “Recently, a series of multi-layer perceptron (MLP) models under the umbrella of “MLP-Mixers" have been proposed in the computer vision domain [19, 25, 27].”
Pay Attention to MLPs (2021, background): “Moreover, the accuracy-parameter/FLOPs tradeoff of gMLPs surpasses all concurrently proposed MLP-like architectures [20, 21, 22], which we attribute to the effectiveness of our Spatial Gating Unit (see Table 3 in the next section for an ablation).”
Vision GNN: An Image is Worth Graph of Nodes (2022, methodology): “The recent neural networks in vision also explore it such as ConvMixer [49], ViT [9] and ResMLP [50].”
MAXIM: Multi-Axis MLP for Image Processing (2022, methodology): “ResMLP [86] adopts an Affine transformation as a substitute to Layer Normalization for acceleration.”
UNeXt: MLP-based Rapid Medical Image Segmentation Network (2022, background): “Recently, MLP-based networks [27,19,14,18] have also been found to be competent in computer vision tasks.”
ResNet strikes back: An improved training procedure in timm (2021, methodology): “In contrast transformers [11, 46] and MLP [43, 44] use AdamW [28] or LAMB optimizer.”
Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results (2022, methodology): “For obtaining top results, a more frequent option is pretraining a model on the larger ImageNet-21K dataset [29], and fine-tuning it on ImageNet-1K [24, 35, 25, 37, 30].”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, positive): “Non-Video Diffusion For non-spatial x that is not video nor images, we use residue MLPs [59] instead of Unet as the backbone for the dynamics model.”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, positive): “The surprising results of ViT on image classification and its scaling ability encourage a lot of follow-up works [16, 59, 61, 62].”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, methodology): “ConvMixer [90] uses up to 9 × 9 convolutions to replace the “mixer” component of ViTs [35] or MLPs [87, 88].”