MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

By Sachin Mehta, Mohammad Rastegari. International Conference on Learning Representations, 2021

2,383 citations recorded by the source corpus.

Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters. Our source code is open-source and available at: https://github.com/apple/ml-cvnets

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/da74a10824193be9d3889ce0d6ed4c6f8ee48b9e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks (2023, methodology): “Second, the attention-based mechanism generally runs slower than its convolutional counterparts and thus becomes less favorable for the current industry [22, 42].”
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications (2022, methodology): “4% top-1 ImageNet-1K classification accuracy which is superior to its recently introduced MobileViT counterpart [29], while requiring 35% less MAdds.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Likewise, MobileViT [18] introduces a hybrid architecture that combines lightweight MobileNet blocks (with point-wise and depth-wise CONV) and MHSA blocks; the former is placed at early stages in the network pipeline to extract low-level features, while the latter is placed in late stages to enjoy t”
SCSA: Exploring the Synergistic Effects Between Spatial and Channel Attention (2024, neutral): “…mainstream blocks: (a) and (b) represent blocks based on the ResNet [19] and its variant series [60]; (c) represents the inverted residual structure based on the MobileNet series [46, 21, 37]; (d) represents the block structure of RepVGG [8], a representative of the reparameterization approach.”
Separable Self-attention for Mobile Vision Transformers (2022, methodology): “1 Introduction Vision transformers (ViTs) [1] have become ubiquitous for a wide variety of visual recognition tasks [2, 3], including mobile vision tasks [4].”
Token Merging: Your ViT But Faster (2022, background): “…et al., 2022; Wang et al., 2020; Bolya et al., 2022), some attempt to prune heads or features (Meng et al., 2022; Voita et al., 2019; Michel et al., 2019), and some attempt to infuse domain-specific modules (Mehta & Rastegari, 2021; Graham et al., 2021; Liu et al., 2021; 2022b; Dong et al., 2022).”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, neutral): “See Figure 5, right.”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, methodology): “MobileViT [28] takes advantage of the success of MobileNets [37] and uses the combination of mobilenet blocks and Transformer blocks to achieve light-weight and low latency.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, methodology): “As shown in Figure 6, MobileOne performs better than MobileViT-S [45] while being 5 × faster on image classification.”
Transformers in Remote Sensing: A Survey (2022, methodology): “the transformers design such as, reduced computational overhead [167], efficient hybrid CNN-transformers backbones [168] and unified architectures for image and video classification [169].”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, methodology): “Two recent works are proposed by designing transformers as convolutions (i.e., MobielViT [68]), and introducing a parallel architecture of MobileNet and ViT (i.e., MobileFormer [13]).”
TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, methodology): “[46] combined standard convolutions and transformers to develop MobileViT, which outperforms the prevailing Mo-bileNets [33] and ShuffleNet [82].”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “For instance, MobileViT [58] achieves better performance than MobileNetV2 while being at least 5× slower on iPhone 12.”
TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation (2022, methodology): “Among these Vision Transformers, MobileViT [31] and Mobile-Former [6] are specially designed for mobile de-”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, methodology): “Motivated by the ability of convolution to capture local information with reduced parameters and computation costs, MobileViT[42] explore combining ViT with convolu-tions, which outperforms light-weight CNN models such as MobileNet-v2/v3[32, 48] with better task-level generalization properties and…”
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, methodology): “• Efficient architectures: Transformers have gone beyond standard language tasks, and have started being used in vision use cases through models like ViT [43] and subsequent optimizations like MobileViT [91].”
EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction (2023, methodology): “To close the gap, MobileViT [38] proposes to combine the strength of CNN and ViT by replacing local processing in convolu-tions with global processing using transformers.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, methodology): “The MobileViT [32] introduces a lightweight vision transformer by combining Transformer blocks with the MobileNetV2 [33] block in series.”
MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, neutral): “MobileViT [35] merges CNN strengths with ViT [13] through global attention blocks.”