Separable Self-attention for Mobile Vision Transformers

By Sachin Mehta, Mohammad Rastegari. Trans. Mach. Learn. Res., 2022

456 citations recorded by the source corpus.

Mobile vision transformers (MobileViT) can achieve state-of-the-art performance across several mobile vision tasks, including classification and detection. Though these models have fewer parameters, they have high latency as compared to convolutional neural network-based models. The main efficiency bottleneck in MobileViT is the multi-headed self-attention (MHA) in transformers, which requires $O(k^2)$ time complexity with respect to the number of tokens (or patches) $k$. Moreover, MHA requires costly operations (e.g., batch-wise matrix multiplication) for computing self-attention, impacting latency on resource-constrained devices. This paper introduces a separable self-attention method with linear complexity, i.e. $O(k)$. A simple yet effective characteristic of the proposed method is that it uses element-wise operations for computing self-attention, making it a good choice for resource-constrained devices. The improved model, MobileViTv2, is state-of-the-art on several mobile vision tasks, including ImageNet object classification and MS-COCO object detection. With about three million parameters, MobileViTv2 achieves a top-1 accuracy of 75.6% on the ImageNet dataset, outperforming MobileViT by about 1% while running $3.2\times$ faster on a mobile device. Our source code is available at: \url{https://github.com/apple/ml-cvnets}

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/066c143b427571fb5568f2c581ea9066478d2e55

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, unknown): “EﬃcientViT [14] and MobileViTv2 [36] introduce self-attention approximations for linear complexity with minor accuracy impacts.”
Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks (2023, methodology): “One notable trend is to pursue a better accuracy-latency trade-off by reducing the complexity of the attention operator [1,29,45,63,68], incorporating convolution into ViTs [6,10,57], or doing both [3,34,49,52].”
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications (2022, methodology): “Our model achieves better accuracy vs compute trade-off compared to SOTA ConvNeXt [25] and MobileViT-V2 [30].”
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, methodology): “Efficient self-attention methods reduce the cost of softmax attention via sparse attention [34, 57, 61, 75] or low-rank approximation [11, 51, 74].”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “The depthwise and pointwise convolutions are employed to learn spatially local representations [36, 37, 39, 53].”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “Many research efforts [58, 59, 60, 47] are taken to alleviate this limitation.”
Mixture-of-Experts in Remote Sensing: A Survey (2026, unknown): “Their model modifies MobileViTV2 [97] to include patch-level MoE layers, where a router clusters patch embeddings and routes each image patch to a small subset of transformer experts, enabling conditional computation within a single network.”