Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

By Qibin Hou, Zihang Jiang, Li Yuan, Mingg-Ming Cheng, Shuicheng Yan, Jiashi Feng. 2021

251 citations recorded by the source corpus.

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies and meanwhile avoid the attention building process in transformers. The outputs are then aggregated in a mutually complementing manner to form expressive representations. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy, greatly improving the performance of recent state-of-the-art MLP-like networks for visual recognition. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. PyTorch/MindSpore/Jittor code is available at https://github.com/Andrew-Qibin/VisionPermutator.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “7 ViP-Large/7 [23] MLP-like 88M 224 224 83.”
DaViT: Dual Attention Vision Transformers (2022, background): “Most recently, MLP-based models [53,54,39,74,8,35,27] draw attention by using multi-layer perceptrons (MLPs) directly over patch features.”
MetaFormer is Actually What You Need for Vision (2021, background): “The follow-up works [25, 34, 50] further improve MLP-like models by data-efficient training and specific MLP module design, gradually narrowing the performance gap to ViT and challenging the dominance of attention as token mixers.”
MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 58, 81], MLP-based [67, 68, 27, 6, 41], convolution-based [15, 19, 80, 56, 78], hybrid [62, 12, 37, 71] or other types of [65, 20] token mixers.”
Sparse-MLP: A Fully-MLP Architecture with Conditional Computation (2021, methodology): “Fully MLP-based models recently achieved promising results in vision (Tolstikhin et al. 2021; Liu et al. 2021a; LeeThorp et al. 2021; Hou et al. 2021).”
Focal Modulation Networks (2022, background): “( ii ) Local-mixing MLPs sample nearby tokens for interactions, using spatial shifting, permutation, and pseudo-kernel mixing [82, 29, 41, 8, 23].”
Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning (2022, background): “In another line, Tolstikhin et al. [52] propose a pure MLP-based architecture, and subsequent papers [20, 33] have interestingly demonstrated that the MLP-based architectures can catch up to transformers.”
Patches Are All You Need? (2022, background): “, 2021a), and vision permutator (Hou et al., 2021), replace one or both blocks with various novel operations.”