Multiscale Vision Transformers

By Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, Christoph Feichtenhofer. 2021

1,678 citations recorded by the source corpus.

We present Multiscale Vision Transformers (MViT) for video and image recognition, by connecting the seminal idea of multiscale feature hierarchies with transformer models. Multiscale Transformers have several channel-resolution scale stages. Starting from the input resolution and a small channel dimension, the stages hierarchically expand the channel capacity while reducing the spatial resolution. This creates a multiscale pyramid of features with early layers operating at high spatial resolution to model simple low-level visual information, and deeper layers at spatially coarse, but complex, high-dimensional features. We evaluate this fundamental architectural prior for modeling the dense nature of visual signals for a variety of video recognition tasks where it outperforms concurrent vision transformers that rely on large scale external pre-training and are 5-10× more costly in computation and parameters. We further remove the temporal dimension and apply our model for image classification where it outperforms prior work on vision transformers. Code is available at: https://github.com/facebookresearch/SlowFast.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Benchmarking Detection Transfer Learning with Vision Transformers (2021, methodology): “We note that recent work, such as Swin [29] and MViT [12], address the single vs . multi-scale feature map problem by modifying the core ViT architecture (in pre-training) so it is inherently multi-scale.”
Early Convolutions Help Transformers See Better (2021, background): “Dominant themes include multi-scale networks [14, 17, 25, 45, 50], increasing depth [42], and locality priors [5, 9, 17, 47, 49].”
Multiview Transformers for Video Recognition (2022, methodology): “In practice, due to the limited amount of training data, applying similar multiscale processing in transformers [11, 21, 44, 74] to reduce complexity of the model has proven to be effective.”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “, Swin [40], MViT [16,32], PVT [53], and PiT [28].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[63] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer.”
The Efficiency Misnomer (2021, methodology): “Despite this, FLOPs has been used as the most common cost indicator in many research papers, especially in the recent computer vision literature, to quantify model efﬁciency (Szegedy et al., 2015; He et al., 2016; Tan and Le, 2019; Feichtenhofer et al., 2019; Fan et al., 2021).”
Transformers in Vision: A Survey (2021, methodology): “Numerous methods have been proposed that make special design choices to perform self-attention more ‘eiciently’, for instance employing pooling/downsampling in self-attention [61, 209, 221], local window-based attention [138, 201], axial-attention [55, 83], low-rank projection attention [192, 208, 2”
A ConvNet for the 2020s (2022, background): “The original ViT [20] ﬁrst studied a hybrid conﬁg-uration, and a large body of follow-up works focused on reintroducing convolutional priors to ViT, either in an explicit [15,16,21,82,86,88] or implicit [45] fashion.”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “Promising results have been obtained for other vision tasks, including image retrieval [23], object detection and semantic segmentation [44, 72, 82, 84], as well as video understanding [2, 7, 24].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
k-means Mask Transformer (2022, background): “Both approaches have boosted various vision tasks, such as image classiﬁcation [19,7,81,44,64,94,32,70,107,103], image generation [78,42], object detection [97,84,81,43,10,116], video recognition [97,19,3,33], semantic segmentation [17,110,46,35,117,115,111,100,11], and panoptic segmentation [94].”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “3.3.”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, methodology): “The latter fuels Multiscale Vision Transformers (MViT) [21], an architecture that extends ViT in a simple way: instead of having a fixed resolution throughout the network, it has a feature hierarchy with multiple stages starting from high-resolution to low-resolution.”
Transformer in Convolutional Neural Networks (2021, methodology): “Similar to PVT [18], MViT [20] utilizes pooling to compute attention on a reduced sequence length.”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including image classification [31, 63, 60], object detection [11, 98, 22], semantic/instance segmentation [86], video understanding [8, 2, 33, 57], point cloud modeling [95, 41], 3D Object Rec”
Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning (2022, background): “- Swin, ViL, TNT, PVT, MViT) in addressing the varying scale / aspect ratios of visual tokens [27, 31, 51, 72, 81].”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “[35], [41], [51], [53], [70], [93], [114] is beneficial for dense prediction tasks such as object detection, instance and semantic segmentation, recent works [21], [23], [32], [38], [42], [54], [58], [66], [77], [78], [95], [104] introduce multi-scale modeling capability by using a pyramid structure”
Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (2021, methodology): “Subsequently, transformer methods have been successfully applied tomanyvisual tasks, including image classification [12], [31], [32], [33], object detection [34], lower-level vision [35], semantic segmentation [36], tracking [37], video instance segmentation [38], image generation [39],multimodal le”