FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization

By Pavan Kumar Anasosalu Vasu, J. Gabriel, Jeff J. Zhu, Oncel Tuzel, Anurag Ranjan. IEEE International Conference on Computer Vision, 2023

384 citations recorded by the source corpus.

The recent amalgamation of transformer and convolutional designs has led to steady improvements in accuracy and efficiency of the models. In this work, we introduce FastViT, a hybrid vision transformer architecture that obtains the state-of-the-art latency-accuracy trade-off. To this end, we introduce a novel token mixing operator, RepMixer, a building block of FastViT, that uses structural reparameterization to lower the memory access cost by removing skip-connections in the network. We further apply traintime overparametrization and large kernel convolutions to boost accuracy and empirically show that these choices have minimal effect on latency. We show that - our model is 3.5× faster than CMT, a recent state-of-the-art hybrid transformer architecture, 4.9× faster than EfficientNet, and 1.9× faster than ConvNeXt on a mobile device for the same accuracy on the ImageNet dataset. At similar latency, our model obtains 4.2% better Top-1 accuracy on ImageNet than MobileOne. Our model consistently outperforms competing architectures across several tasks - image classification, detection, segmentation and 3D mesh regression with significant improvement in latency on both a mobile device and a desktop GPU. Furthermore, our model is highly robust to out-of-distribution samples and corruptions, improving over competing robust models. Code and models are available at https://github.com/apple/ml-fastvit

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/52cc542b60c99e8866a7fc515c10f443089005f2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zijia Lin
2. Hui Chen
3. Ao Wang
4. Chas Leichner
5. Ying Shan
6. Yixiao Ge
7. Yiyuan Zhang
8. Lin Song

In Computer Vision and Pattern Recognition

1. Zijia Lin
2. Hui Chen
3. Ao Wang
4. Colby R. Banbury
5. Chas Leichner
6. Ying Shan
7. Yixiao Ge
8. Berkin Akin

Citations in the corpus

• UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition (2023, methodology): “…ADE20K semantic segmentation [98], and COCO object detection [44], outperforming the existing large-kernel ConvNets such as RepLKNet [19], SLaK [48], and recent powerful architectures including ConvNeXt V2 [85], FastViT [77], Swin V2 [51] and DeiT III [75] in terms of both accuracy and efficiency.”
• MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, positive): “We benchmark our models against the leading efficient models, including hybrid (MiT-EfficientViT [14], FastViT [45], NextViT [28]) and convolutional models (MobileOne [46], ConvNext [34], and previous MobileNets [21] [38] [20]) based on their reported Top-1 Accuracies and our latency evaluations.”
• Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “Following [35, 36, 48, 59], we train all models from scratch for 300 epochs or 450 epochs using the same training recipe.”