EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction
By Han Cai, Junyan Li, Muyan Hu, Chuang Gan, Song Han. IEEE International Conference on Computer Vision, 2023
296 citations recorded by the source corpus.
High-resolution dense prediction enables many appealing real-world applications, such as computational photography, autonomous driving, etc. However, the vast computational cost makes deploying state-of-the-art high-resolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel lightweight multi-scale attention. Unlike prior high-resolution dense prediction models that rely on heavy self-attention, hardware-inefficient large-kernel convolution, or complicated topology structure to obtain good performances, our lightweight multi-scale attention achieves a global receptive field and multi-scale learning (two critical features for high-resolution dense prediction) with only lightweight and hardware-efficient operations. As such, EfficientViT delivers remarkable performance gains over previous state-of-the-art high-resolution dense prediction models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, our EfficientViT provides up to 8.8× and 3.8× GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, EfficientViT provides up to 6.4× speedup over Restormer while providing 0.11dB gain in PSNR.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/9c53679e7c2107e36e6c60c19464c607bf3459cd
Citations in the corpus
- SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, unknown): “Recent work such as Q-Diffusion (Li et al., 2023) has achieved high-quality generation using only 4-bit weights.”
- MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, unknown): “EfficientViT [14] and MobileViTv2 [36] introduce self-attention approximations for linear complexity with minor accuracy impacts.”
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, unknown): “The computational complexity of their attention layers prohibits na¨ıvely scaling them to higher resolutions and several works proposed alternatives (Zhu et al., 2021; Liu et al., 2021; Li et al., 2022c; Chu et al., 2021; Liu et al., 2022a; 2023; Cai et al., 2023; Jaegle et al., 2022).”