MetaFormer is Actually What You Need for Vision

By Weihao Yu, Romy Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan. 2021

1,389 citations recorded by the source corpus.

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1 % top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 49%/61% fewer MACs. The effectiveness of Pool-Former verifies our hypothesis and urges us to initiate the concept of “MetaFormer”, a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Learning with Recoverable Forgetting (2022, positive): “It has been successfully conducted in deep model compression [61], incremental learning [37], continual learning [25,57] and other tasks other than classification [5,53,22,50,36,60,51,55,23].”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “The meta-architecture of Gated CNN and Mamba can be considered as a simplified integration of the MetaFormer’s [99] token mixer and an MLP, akin to MetaNeXt [101].”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, methodology): “Following this direction, Dosovitskiy et al. [11] first employ a transformer in the domain of computer vision and introduce a new architecture paradigm, ViT, which achieves promising results [64, 48].”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “Fig 4 illustrates an LSKNet Block, which is inspired by ConvNeXt [41], MetaFormer [81], PVT-v2 [63], Conv2Former [28] and VAN [19].”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, background): “(2) From the architecture view [9, 22, 23], besides MHSA, ViTs contain a series of advanced components that are not included in standard CNNs, such as Layer Normalization (LN) [24], feed-forward network (FFN) [1], GELU [25], etc.”
EMCAD: Efficient Multi-Scale Convolutional Attention Decoding for Medical Image Segmentation (2024, background): “Hierarchical vision transformers like Swin [34], PVT [55, 56], MaxViT [49], MERIT [43], ConvFormer [33], and MetaFormer [59] have been introduced to further improve the performance in this field.”
DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, neutral): “, Resnet, Transformers, Metaformer [12, 38, 46]).”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “MetaFormer [103] suggests pooling layer is an alternate to self-attention.”
How Do Vision Transformers Work? (2022, background): “The most widely accepted explanation for the success of MSAs is their weak inductive bias and capture of long-range dependencies (See, e.g., (Dosovitskiy et al., 2021; Naseer et al., 2021; Tuli et al., 2021; Yu et al., 2021; Mao et al., 2021; Chu et al., 2021)).”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “Later research further improves ViTs by incorporating hierarchical design [77, 74], injecting locality with the aid of convolutions [26, 18, 28, 68], or exploring different types of token mixing such as local attention [52, 20], spatial MLP mixer [71, 70], and non-parameterized pool mixer [86].”
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, background): “MetaFormer [91] suggested large-kernel pooling layer was an alternative to self-attention.”
RWKV: Reinventing RNNs for the Transformer Era (2023, methodology): “FFN with R gate Prior research (Tolstikhin et al., 2021; Liu et al., 2021; Yu et al., 2022) suggests that self-attention may not be as essential in Transformer-based vision tasks as previously thought.”