TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation

By Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, Chunhua Shen. Computer Vision and Pattern Recognition, 2022

301 citations recorded by the source corpus.

Although vision transformers (ViTs) have achieved great success in computer vision, the heavy computational cost hampers their applications to dense prediction tasks such as semantic segmentation on mobile devices. In this paper, we present a mobile-friendly architecture named Token Pyramid Vision Transformer (TopFormer). The proposed TopFormer takes Tokens from various scales as input to produce scale-aware semantic features, which are then in-Jected into the corresponding tokens to augment the representation. Experimental results demonstrate that our method significantly outperforms CNN- and ViT-based networks across several semantic segmentation datasets and achieves a good trade-off between accuracy and latency. On the ADE20K dataset, TopFormer achieves 5% higher accuracy in mIoU than MobileNetV3 with lower latency on an ARM-based mobile device. Furthermore, the tiny version of TopFormer achieves real-time inference on an ARM-based mobile device with competitive results. The code and models are available at: https://github.com/hustvl/TopFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f634a09747e4ca11754a9bfdccf7485c884f9f86

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “, Fully Convolutional Networks [29], U-net [38], DeepLab family [4–6] and more recently Vision Transformer based methods [16, 53, 58].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “TopFormer [328] proposes a token pyramid module that takes the tokens from various scales as input to produce the scale-aware semantic feature.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, background): “Building on the competitive performance of ViTs, subsequent works have sought to incorporate spatial inductive biases to enhance their stability and performance [8,17], design more efficient self-attention operations by regularizing the pattern of attention mechanisms [13, 69], and adapt ViTs to a d”
Deep Dual-Resolution Networks for Real-Time and Accurate Semantic Segmentation of Traffic Scenes (2023, background): “The recent TopFormer [37] for mobile semantic segmentation is a encoder-decoder architecture built upon light-weight transformer backbones.”
Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism (2023, background): “Built upon the concept of global information fusion, TopFormer [45] has achieved remarkable results in semantic segmentation tasks.”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “Inspired by the competitive performance and global receptive ﬁeld of transformer models, follow-up works are proposed to reﬁne the architecture [36, 37], explore the relationship between CONV nets and ViT [38, 39, 40], and adapt ViT to different computer vision tasks [13, 41, 42, 43, 44, 45, 46].”
YOLOv10: Real-Time End-to-End Object Detection (2024, background): “Self-attention [58] is widely employed in various visual tasks due to its remarkable global modeling capability [38, 14, 76].”