Rethinking Vision Transformers for MobileNet Size and Speed

By Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, Jian Ren. IEEE International Conference on Computer Vision, 2022

327 citations recorded by the source corpus.

With the success of Vision Transformers (ViTs) in computer vision tasks, recent arts try to optimize the performance and complexity of ViTs to enable efficient deployment on mobile devices. Multiple approaches are proposed to accelerate attention mechanism, improve inefficient designs, or incorporate mobile-friendly lightweight convolutions to form hybrid architectures. However, ViT and its variants still have higher latency or considerably more parameters than lightweight CNNs, even true for the years-old MobileNet. In practice, latency and size are both crucial for efficient deployment on resource-constraint hardware. In this work, we investigate a central question, can transformer models run as fast as MobileNet and maintain a similar size? We revisit the design choices of ViTs and propose a novel supernet with low latency and high parameter efficiency. We further introduce a novel fine-grained joint search strategy for transformer models that can find efficient architectures by optimizing latency and number of parameters simultaneously. The proposed models, EfficientFormerV2, achieve 3.5% higher top-1 accuracy than MobileNetV2 on ImageNet-1K with similar latency and parameters. This work demonstrate that properly designed and optimized vision transformers can achieve high performance even with MobileNet-level size and speed 1.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/f35016b3180808fa97d59acbdecf47d6e2ed2819

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zijia Lin
2. Hui Chen
3. Ao Wang
4. Hengjun Pu
5. Guiguang Ding
6. Dacheng Tao
7. Fumin Shen
8. Cheng Deng

Citations in the corpus

• Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “Many efficient design principles have been proposed to enhance the computational efficiency of ViTs for mobile devices [5, 35, 46, 49].”