MPViT: Multi-Path Vision Transformer for Dense Prediction

By Youngwan Lee, Jonghee Kim, Jeffrey Willette, Sung Ju Hwang. Computer Vision and Pattern Recognition, 2021

361 citations recorded by the source corpus.

Dense computer vision tasks such as object detection and segmentation require effective multi-scale feature representation for detecting or classifying objects or regions with varying sizes. While Convolutional Neural Networks (CNNs) have been the dominant architectures for such tasks, recently introduced Vision Transformers (ViTs) aim to replace them as a backbone. Similar to CNNs, ViTs build a simple multi-stage structure (i.e., fine-to-coarse) for multi-scale representation with single-scale patches. In this work, with a different perspective from existing Transformers, we explore multi-scale patch embedding and multi-path structure, constructing the Multi-Path Vision Transformer (MPViT). MPViT embeds features of the same size (i.e., sequence length) with patches of different scales simultaneously by using overlapping convolutional patch embedding. Tokens of different scales are then independently fed into the Transformer encoders via multiple paths and the resulting features are aggregated, enabling both fine and coarse feature representations at the same feature level. Thanks to the diverse, multi-scale feature representations, our MPViTs scaling from tiny (5M) to base (73M) consistently achieve superior performance over state-of-the-art Vision Transformers on ImageNet classification, object detection, instance segmentation, and semantic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/15b0e710a9b8069d898ae6a0963d627e0fb86bd8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Focal Modulation Networks (2022, background): “Meanwhile, a number of models have been proposed by augmenting SA with (depth-wise) convolutions to capture long-range dependencies with a good awareness of local structures [74, 22, 78, 20, 18, 35, 7, 17].”
Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, neutral): “MonoViT [43] uses MPViT [23] as its encoder and has achieved state-of-the-art accuracy.”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, methodology): “Then, considering the advantage of ViT at building global dependencies while shows limitations modeling local details [25], extracted tokens are processed through both convolutional layers and Transformers blocks, in a parallel and complementary manner – i.e. , using the four branches shown in Fig.”
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, neutral): “Another line of work modified the ViT architecture to produce a hierarchy of features (Fan et al., 2021; Xie et al., 2021; Yuan et al., 2021; Ranftl et al., 2021; Chen et al., 2021; Lee et al., 2022).”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Motivated by MViT, from the architecture level, MPViT [133] introduces multiscale patch embedding and multi-path structure to explore tokens of different scales jointly.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, methodology): “Thus, some tailored encoders appear, including Res2Net [20], HRNet [71], SETR [96], SegFormer [80], HRFormer [88], MPViT [38], DPT [63], etc .”
Decoupling Deep Learning for Interpretable Image Recognition (2022, background): “With the continuous development of neural networks (NNs) [24, 28, 11, 18, 16, 19], their interpretability is a research direction that has received extensive attention.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, background): “Following this trend, various architectures of dense prediction transformers [48], [49], [50] and semantic segmentation transformers [51], [52], [53] emerge in the field.”
OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “Inspired by recent advances that introduce locality into the transformer [49, 56, 24], we also design the encoder as a hybrid structure.”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “[61], [66], [78], [89], [91] introduce multi-scale information in patch embedding layers [78] or self-attention blocks [42], [66], [91] or add extra branches [10], [61], [89] to perform convolution operation.”