SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

By Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, José Manuel Álvarez, Ping Luo. 2021

8,811 citations recorded by the source corpus.

We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C. Code will be released at: github.com/NVlabs/SegFormer.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, background): “0 SegFormer-B5 [62] SegFormer ImgNet-1k 84.”
FocalClick: Towards Practical Interactive Image Segmentation (2022, methodology): “In this paper, we conduct experiments *https://github.com/Algy/fast-slic Algorithm 1 Simulate Defective Mask using Super-Pixels Require: Image, GTMask, maxIOU=0.85, minIOU=0.75 SimMask← GT while True do ErrorType← Rand([Boundary, External, Internal]) PixelNumber← Rand([50, 100, 200, 300, 500, 700]) ”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “For semantic segmentation, Mask2Former with Swin-B backbone outperforms the state-of-the-art SegFormer [59].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Transformers in Vision: A Survey (2021, background): “SegFormer [101] has a hierarchical pyramid ViT [93] (without position encoding) as an encoder, and a simple”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, methodology): “Inspired by the success of Transformers in NLP [20,76], the Vision Transformer (ViT) [22] was recently proposed and has been successfully applied to multiple computer vision tasks, including image classification [44, 72, 73, 86], object detection [44, 79, 88], semantic segmentation [44, 82, 91] and ”
Continual Segmentation with Disentangled Objectness Learning and Class Recognition (2024, positive): “Mask-Former [9] was the first to introduce mask classification in semantic segmentation, which was dominated by per-pixel classification paradigm via convolutional nets [8, 46, 52] or Transformers [42, 53].”
End-to-End Instance Edge Detection (2022, methodology): “To address this, researchers have proposed methods [68,60,44] that reduce both computation and memory complexity, allowing the transformer to perform dense prediction tasks [63,12,54,59].”
Shunted Self-Attention via Multi-Scale Token Aggregation (2021, result): “We also take SegFormer [28] as the framework and compare our backbone with the MiT in the SegFormer.”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, methodology): “Besides, transformer-based models and the variants have also been proposed in object detection [7], semantic segmentation [64], image generation [30], etc.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “Given an input image, AdaViT learns to adaptively derive policies on which patches , self-attention heads and transformer blocks to use or activate in the transformer backbone conditioned on the input image, encouraging us-ing less computation while maintaining the classiﬁcation accuracy.”
SegDiff: Image Segmentation with Diffusion Probabilistic Models (2021, methodology): “The leading methods in the ﬁeld employ encoder-decoder networks of varied architectures [4, 31, 38, 50, 52, 53].”
TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation (2022, methodology): “SegFormer [46] achieves great performance (37.”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “Presently, the transformer has been adopted in numerous vision tasks, such as image recognition [17, 60], segmentation [62, 69, 87, 49], object detection [5, 89], and image restoration [7, 38, 74, 36, 8].”
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers (2021, methodology): “We use the SegFormer-B3 backbone which is a hiearchical vision transformer Xie et al. (2021).”