MetaFormer Baselines for Vision
By Weihao Yu, Chenyang Si, Pan Zhou, Romy Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, Xinchao Wang. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
348 citations recorded by the source corpus.
MetaFormer, the abstracted architecture of Transformer, has been found to play a significant role in achieving competitive performance. In this paper, we further explore the capacity of MetaFormer, again, by migrating our focus away from the token mixer design: we introduce several baseline models under MetaFormer using the most basic or common mixers, and demonstrate their gratifying performance. We summarize our observations as follows: <list list-type="ordered"> <list-item><label>1)</label><p><italic>MetaFormer ensures solid lower bound of performance:</italic> By merely adopting identity mapping as the token mixer, the MetaFormer model, termed <italic>IdentityFormer</italic>, achieves <inline-formula><tex-math notation="LaTeX">$> $</tex-math><alternatives><mml:math><mml:mo>></mml:mo></mml:math><inline-graphic xlink:href="wang-ieq1-3329173.gif"/></alternatives></inline-formula>80% accuracy on ImageNet-1 K.</p></list-item> <list-item><label>2)</label><p><italic>MetaFormer works well with arbitrary token mixers:</italic> When specifying the token mixer as even a random matrix to mix tokens, the resulting model <italic>RandFormer</italic> yields an accuracy of <inline-formula><tex-math notation="LaTeX">$> $</tex-math><alternatives><mml:math><mml:mo>></mml:mo></mml:math><inline-graphic xlink:href="wang-ieq2-3329173.gif"/></alternatives></inline-formula>81%, outperforming IdentityFormer. Rest assured of MetaFormer's results when new token mixers are adopted.</p></list-item> <list-item><label>3)</label><p><italic>MetaFormer effortlessly offers state-of-the-art results:</italic> With just conventional token mixers dated back five years ago, the models instantiated from MetaFormer already beat state of the art.</p></list-item> </list> <list list-type="ordered"> <list-item><label>a)</label><p><italic>ConvFormer outperforms ConvNeXt:</italic> Taking the common depthwise separable convolutions as the token mixer, the model termed <italic>ConvFormer</italic>, which can be regarded as pure CNNs, outperforms the strong CNN model ConvNeXt.</p></list-item> <list-item><label>b)</label><p><italic>CAFormer sets new record on ImageNet-1 K:</italic> By simply applying depthwise separable convolutions as token mixer in the bottom stages and vanilla self-attention in the top stages, the resulting model <italic>CAFormer</italic> sets a new record on ImageNet-1 K: it achieves an accuracy of 85.5% at <inline-formula><tex-math notation="LaTeX">$224 \times 224$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>224</mml:mn><mml:mo>×</mml:mo><mml:mn>224</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="wang-ieq3-3329173.gif"/></alternatives></inline-formula> resolution, under normal supervised training without external data or distillation.</p></list-item> </list> In our expedition to probe MetaFormer, we also find that a new activation, <italic>StarReLU</italic>, reduces 71% FLOPs of activation compared with commonly-used GELU yet achieves better performance. Specifically, StarReLU is a variant of Squared ReLU dedicated to alleviating distribution shift. We expect StarReLU to find great potential in MetaFormer-like models alongside other neural networks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/fc4cbc7a75f5a3bbca59db5513231555f078fe78
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “The block structure of lightweight ViTs [35, 36, 47] incorporates an important design feature, namely the separate token mixer and channel mixer [70].”
- InceptionNeXt: When Inception Meets ConvNeXt (2023, positive): “ConvNeXt is not alone: More and more works have shown similar observations [14, 18, 24, 38, 51, 68, 76, 78], like RepLKNet [14] and SLaK [38].”
- MambaOut: Do We Really Need Mamba for Vision?* (2024, positive): “Formally, given the input X ∈ R N × D , the meta-architecture is formulated as: (9) where Norm( · ) represents normalization [2, 42, 91]; TokenMixer( · ) refers to the module to conduct token mixing [100]; are learnable parameters with MLP expansion r ; σ is activation function [27, 37].”