Mobile-Former: Bridging MobileNet and Transformer

By Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, Zicheng Liu. Computer Vision and Pattern Recognition, 2021

695 citations recorded by the source corpus.

We present Mobile-Former, a parallel design of MobileNet and transformer with a two-way bridge in between. This structure leverages the advantages of MobileNet at local processing and transformer at global interaction. And the bridge enables bidirectional fusion of local and global features. Different from recent works on vision transformer, the transformer in Mobile-Former contains very few tokens (e.g. 6 or fewer tokens) that are randomly initialized to learn global priors, resulting in low computational cost. Combining with the proposed light-weight cross attention to model the bridge, Mobile-Former is not only computationally efficient, but also has more representation power. It outperforms MobileNetV3 at low FLOP regime from 25M to 500M FLOPs on ImageNet classification. For instance, Mobile-Former achieves 77.9% top-1 accuracy at 294M FLOPs, gaining 1.3% over MobileNetV3 but saving 17% of computations. When transferring to object detection, Mobile-Former outperforms MobileNetV3 by 8.6 AP in RetinaNet framework. Furthermore, we build an efficient end-to-end detector by replacing backbone, encoder and decoder in DETR with Mobile-Former, which outperforms DETR by 1.3 AP but saves 52% of computational cost and 36% of parameters. Code will be released at https://github.com/aaboys/mobileformer.

Subfields: Computer Vision and Pattern Recognition, Neurology

View Paper: https://www.semanticscholar.org/paper/a66686e60a3eda0c606e036403cf0a07a5962595

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, neutral): “5 3,262 224 Trans Mobile-Former-214M [14] 76.”
Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks (2023, methodology): “Apart from the above pure convolutional neural networks (CNNs), there is an emerging interest in making vision transformers (ViTs) [12] and multilayer perceptrons (MLPs) architectures [64] smaller and faster.”
BEVT: BERT Pretraining of Video Transformers (2021, background): “And the recent work Mobile-Former [12] further extends transformer at low FLOP regime.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “Although reducing the model size of ViT models to match the constraints of mobile devices is possible, their performance is often suboptimal compared to lightweight CNNs [3].”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “In application scenarios demanding high efficiency, MobileFormer [5] maintains two paths for convolution and Transformer respectively and enjoys the benefit from both modules.”
Transfer learning for medical image classification: a literature review (2022, neutral): “[63] proposed a novel architecture that is a parallel design of MobileNet and ViT, in view of achieving not only more efficient computation but also better model performance.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “For instance, some efforts consider designing new architectures or operations by changing the linear layers with convolutional layers (CONV) [23], combining self-attention with MobileNet blocks [24], or introducing sparse attention [25, 26, 27], to reduce the computational cost, while other efforts ”
Fast Vision Transformers with HiLo Attention (2022, background): “Note that this paper does not consider mobile-level architectures [6, 28].”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, background): “Subsequent works (e.g., Graham et al., 2021; Dai et al., 2021; Liu et al., 2021; Wang et al., 2021; Yuan et al., 2021b; Chen et al., 2021b) shows that this substandard optimizability is due to the lack of spatial inductive biases in ViTs.”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, background): “Moreover, some works [10], [27], [61], [69], [82], [85], [96] combine CNNs and Transformers”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “Mobile-Former [34] takes a bidirectional bridge between CNN and transformer to leverage the advantage of global and local concepts.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, methodology): “Furthermore, we note the convolutional models such as MobileNets [43, 47, 56] have lower latency for similar FLOPs and parameter count than their transformer counter-parts [5,45,58].”
EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction (2023, methodology): “Mobile-Former [9] proposes to parallelize MobileNet and Trans-former with a two-way bridge in between for feature fusing.”
MobileNetV4 - Universal Models for the Mobile Ecosystem (2024, neutral): “MobileFormer [7] parallelizes a MobileNet and a Transformer with a two-way bridge in between for feature fusing.”
Separable Self-attention for Mobile Vision Transformers (2022, methodology): “However, on GPU, both MobileFormer and MobileViTv2 run at a similar speed.”