EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

By Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang, Han Hu, Yixuan Yuan. Computer Vision and Pattern Recognition, 2023

807 citations recorded by the source corpus.

Vision transformers have shown great success due to their high model capabilities. However, their remarkable performance is accompanied by heavy computation costs, which makes them unsuitable for real-time applications. In this paper, we propose a family of high-speed vision transformers named Efficient ViT. We find that the speed of existing transformer models is commonly bounded by memory inefficient operations, especially the tensor reshaping and element-wise functions in MHSA. Therefore, we design a new building block with a sandwich layout, i.e., using a single memory-bound MHSA between efficient FFN layers, which improves memory efficiency while enhancing channel communication. Moreover, we discover that the attention maps share high similarities across heads, leading to computational redundancy. To address this, we present a cascaded group attention module feeding attention heads with different splits of the full feature, which not only saves computation cost but also improves attention diversity. Comprehensive experiments demonstrate EfficientViT outperforms existing efficient models, striking a good trade-off between speed and accuracy. For instance, our EfficientViT-M5 surpasses MobileNetV3-Large by 1.9% in accuracy, while getting 40.4% and 45.2% higher throughput on Nvidia V100 GPU and Intel Xeon CPU, respectively. Compared to the recent efficient model MobileViT-XXS, EfficientViT-M2 achieves 1.8% superior accuracy, while running $5.8\times/3.7\times$ faster on the GPU/CPU, and $7.4\times faster$ when converted to ONNX format. Code and models are available at here.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/9a83aeadc8db65fb6da39ec977360541cddaff5c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Raghuraman Krishnamoorthi

Citations in the corpus

MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, neutral): “MNv4-Hybrid models were excluded from CoreML evaluation due to the lack of PyTorch implementation of Mobile MQA. tleneck (IB), ConvNext [34], and FFN [13] - and introduces the Extra Depthwise (ExtraDW) IB block.”
U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation (2024, background): “Although highly sophisticated, transformers tend to overfit when dealing with limited datasets, indicating their data-hungry nature [50, 79].”
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything (2023, methodology): “This trick has been used in many follow-up works including LeViT[22], EfficientFormer[35], Next-ViT[33], Tiny-ViT[61], Castling-ViT[66], EfficientViT[38].”