Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

By Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao A. Xiang, Philip H. S. Torr and 1 more. 2020

3,740 citations recorded by the source corpus.

Most recent semantic segmentation methods adopt a fully-convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces the spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focused on increasing the receptive field, through either dilated/atrous convolutions or inserting attention modules. However, the encoder-decoder based FCN architecture remains unchanged. In this paper, we aim to provide an alternative perspective by treating semantic segmentation as a sequence-to-sequence prediction task. Specifically, we deploy a pure transformer (i.e., without convolution and resolution reduction) to encode an image as a sequence of patches. With the global context modeled in every layer of the transformer, this encoder can be combined with a simple decoder to provide a powerful segmentation model, termed SEgmentation TRansformer (SETR). Extensive experiments show that SETR achieves new state of the art on ADE20K (50.28% mIoU), Pascal Context (55.83% mIoU) and competitive results on Cityscapes. Particularly, we achieve the first position in the highly competitive ADE20K test server leaderboard on the day of submission.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “3 mIoU by SETR [73] which has a larger model size).”
Visual Prompt Tuning (2022, methodology): “We explore the feasibility of VPT beyond visual classification, by evaluating ADE20K [90] semantic segmentation task with a Transformer model, SETR-PUP [89].”
EDTER: Edge Detection with Transformer (2022, methodology): “5%, 3%, 3%) by BiMLA compared with SETR-MLA [72] in both stages.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “…[31] Swin-L † 65.9 - - - - Panoptic-DeepLab [11] SWideRNet [9] 66.4 40.1 82.2 - - Panoptic-DeepLab [11] SWideRNet [9] 67.5 ∗ 43.9 ∗ 82.9 ∗ - - SETR [64] ViT-L † nal predictions of Mask2Former after the Transformer de-coder layers, i.e ., layer 9, and AR@100 consistently improves with more decoder…”
Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, methodology): “Note, that any per-pixel classificationbased segmentation model fits the pixel-level module design including recent Transformer-based models [37, 53, 29].”
K-Net: Towards Unified Image Segmentation (2021, methodology): “In Table 3b, we further compare K-Net with other state-of-the-art methods [9, 69] with test-time augmentation on the validation set.”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Transformers in Vision: A Survey (2021, methodology): “Below, we discuss axial self-attention [205], a crossmodal approach [231] that can segment regions corresponding to a given language expression, and ViTs-based segmentation architectures [180, 221, 250].”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “Concurrent work extends this work to video classification [2, 6] and semantic segmentation [35, 67].”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “[7] proposed SETR to demonstrate the feasibility of using Transformers in this task.”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “Since then, the pure Transformer design has been adopted to various vision tasks including semantic segmentation [114], point cloud classification [113], action recoginition [10, 83, 6].”
ViViT: A Video Vision Transformer (2021, background): “ViT has inspired a large amount of follow-up work in the community, and we note that there are a number of concurrent approaches on extending it to other tasks in computer vision [71, 74, 84, 85] and improving its data-efﬁciency [64, 48].”
Transformer in Transformer (2021, background): “Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
DeepViT: Towards Deeper Vision Transformer (2021, methodology): “Motivated by their success on the NLP tasks, recent researchers attempted to combine the self-attention mechanism into CNNs for computer vision tasks [40, 3, 4, 32, 24, 50, 49].”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “Also for vision, transformer-based architectures have shown great capabilities and potentials on image classification [13], object detection [5], and semantic segmentation [69].”