MViTv2: Improved Multiscale Vision Transformers for Classification and Detection

By Yanghao Li, Chaoxia Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer. 2021

962 citations recorded by the source corpus.

In this paper, we study Multiscale Vision Transformers (MViTv2) as a unified architecture for image and video classification, as well as object detection. We present an improved version of MViT that incorporates decomposed relative positional embeddings and residual pooling connections. We instantiate this architecture in five sizes and evaluate it for ImageNet classification, COCO detection and Kinetics video recognition where it outperforms prior work. We further compare MViTv2s' pooling attention to window attention mechanisms where it outperforms the latter in accuracy/compute. Without bells-and-whistles, MViTv2 has state-of-the-art performance in 3 domains: 88.8% accuracy on ImageNet classification, 58.7 APbox on COCO object detection as well as 86.1% on Kinetics-400 video classification. Code and models are available at https://github.com/facebookresearch/mvit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “Therefore we adopt the hybrid approach from [26] that includes four global self-attention blocks placed evenly at each d/4th block (these coincide with the up-/downsampling locations used for FPN integration; see Figure 1).”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, methodology): “To focus on comparing backbones under as fair conditions as possible, we incorporate the Swin [40] and MViTv2 [32] backbones into our implementation.”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale (2022, neutral): “7 MViTv2-H [58] 667M IN-21K (14M) 5122 88.”
ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, background): “Window-based attention with relative position embedding [25, 26] has been explored to alleviate the heavy memory burden of dealing with the higher resolution feature maps.”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks (2023, methodology): “We adopt a minimalist setting, i.e . conducting a single-stage SFT using only the LLaVA-Mix-visual 665K [64].”
PETR: Position Embedding Transformation for Multi-View 3D Object Detection (2022, background): “In transformer, the features are usually added with position embedding, which provides the position information of the sequence [15,47,11,10,54], image [13,53,27] and video [1,24,52].”
DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
InceptionNeXt: When Inception Meets ConvNeXt (2023, positive): “Many ViT variants [15, 36, 39, 64, 67, 75, 79], like DeiT [64] and Swin [39], are proposed and have achieved remarkable performance across a wide range of vision tasks.”
Generative artificial intelligence: a systematic review and applications (2024, positive): “Advancing further, Li et al. [76] introduced Multiscale Vision Transformers (MViTv2) as a comprehensive architectural framework suitable for tasks encompassing image and video classification, along with object detection.”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, methodology): “, using downsmapled features to produce attention weights like [34]), and more advanced training methods and hybrid architectures (e.”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
DETRs with Hybrid Matching (2022, neutral): “MViTv2 [27] Cascade Mask R-CNN MViTv2-L 1333× 800 50 55.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, background): “They have demonstrated superior performance compared to CNNs on a variety of vision tasks, like image classification [39, 63], semantic segmentation [6, 66] and object detection [4, 34].”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, positive): “Also, our X-CLIP-L/14 8 f outperforms MViTv2-L [27] by +1 .”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, background): “Following this success, several efforts explore the potential of ViT by improving training strategies [3, 4, 5], introducing architecture changes [6, 7], redesigning attention mechanisms [8, 9], and elevating the performance of various vision tasks such as classification [10, 11, 12], segmentation [”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, neutral): “The architecture of InternVL3 follows the same general framework as its predecessors, adhering to the “ViT-MLP-LLM” paradigm [66, 18, 41, 20].”