MaxViT: Multi-Axis Vision Transformer

By Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, A. Bovik, Yinxiao Li. European Conference on Computer Vision, 2022

1,088 citations recorded by the source corpus.

Transformers have recently gained significant attention in the computer vision community. However, the lack of scalability of self-attention mechanisms with respect to image size has limited their wide adoption in state-of-the-art vision backbones. In this paper we introduce an efficient and scalable attention model we call multi-axis attention, which consists of two aspects: blocked local and dilated global attention. These design choices allow global-local spatial interactions on arbitrary input resolutions with only linear complexity. We also present a new architectural element by effectively blending our proposed attention model with convolutions, and accordingly propose a simple hierarchical vision backbone, dubbed MaxViT, by simply repeating the basic building block over multiple stages. Notably, MaxViT is able to ''see'' globally throughout the entire network, even in earlier, high-resolution stages. We demonstrate the effectiveness of our model on a broad spectrum of vision tasks. On image classification, MaxViT achieves state-of-the-art performance under various settings: without extra data, MaxViT attains 86.5% ImageNet-1K top-1 accuracy; with ImageNet-21K pre-training, our model achieves 88.7% top-1 accuracy. For downstream tasks, MaxViT as a backbone delivers favorable performance on object detection as well as visual aesthetic assessment. We also show that our proposed model expresses strong generative modeling capability on ImageNet, demonstrating the superior potential of MaxViT blocks as a universal vision module. The source code and trained models will be available at https://github.com/google-research/maxvit.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2ad12a7be5eaf339a98c4defd8669e11fe726acc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “For reducing the computation complexity of ViTs, many works propose new modules and architecture design [39, 28, 22, 43, 63], while others eliminate redundancies in attention mechanism [75, 31, 10, 46, 15, 62, 72, 5].”
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer (2022, methodology): “To ameliorate this issue, numerous methods have introduced locality into self-attention, such as Swin [25], CSwin [7], Twins [6], window [40,34], and sparse attention [37,35,43].”
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, neutral): “6 MaxViT-XL [95] 475M IN-21K (14M) 5122 88.”
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders (2023, result): “We compare our results to the state-of-the-art architecture designs, including convolutionbased [33,40], transformer-based [8], and hybrid designs [6, 42].”
TransNeXt: Robust Foveal Visual Perception for Vision Transformers (2023, neutral): “Moreover, under similar throughput conditions, our model demonstrates competitive top-1 accuracy on ImageNet-1K compared to previous state-of-the-art models such as MaxViT [25], BiFormer [34], and QuadTree [24].”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
Dual Aggregation Transformer for Image Super-Resolution (2023, background): “Transformer demonstrates remarkable performance in high-level vision tasks [13, 39, 37].”
MetaFormer Baselines for Vision (2022, methodology): “The power of MetaFormer can also be verified by the recent models adopting MetaFormer as the general architecture but with different attention-based [40, 57, 70], MLP-based [66, 67, 27, 6, 41], convolution-based [15, 19, 79, 56, 77], hybrid [61, 12, 37, 70] or other types of [64, 20] token mixers.”
EMCAD: Efficient Multi-Scale Convolutional Attention Decoding for Medical Image Segmentation (2024, methodology): “Hierarchical vision transformers like Swin [34], PVT [55, 56], MaxViT [49], MERIT [43], ConvFormer [33], and MetaFormer [59] have been introduced to further improve the performance in this field.”
Dilated Neighborhood Attention Transformer (2022, methodology): “More recently, MaxViT [37] explored a hybrid model, which uses a combination of Window Self Attention [26] (local self attention), Grid Attention (sparse self attention), and convolutions.”
SVDiff: Compact Parameter Space for Diffusion Fine-Tuning (2023, background): “Recent advancements have also explored transformer-based architectures [67, 44, 6, 7].”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “ViT [22]/DeiT [73] × APE × X × × × PVT [77] × × APE × X × X × Swin [54] × × RPE X × × X × Twins [13] × × CPE X X × X × GG [94] × × RPE/APE × × X X × Shuffle [38] X × RPE X × X X × MaxViT [75] X X CPE X × X X × CrossFormer [78] X X RPE X × X X X ViL [104] × × RPE/APE X X × X × NAT [32] X X RPE X × × ”
CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers (2022, methodology): “To attain such local-global properties efficiently, we propose a sparse 3D attention model called fused axial attention (FAX), which performs both local window-based attention and sparse global interactions, inspired by [32, 48, 49].”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, methodology): “Although the idea of solving CV tasks by SM is straightforward, it serves as the foundation to some of the best-performing CV algorithms [38, 41, 39].”