Segmenter: Transformer for Semantic Segmentation

By Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, Cordelia Schmid. 2021

1,968 citations recorded by the source corpus.

Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embed-dings with a point-wise linear decoder or a mask trans-former decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

VOLO: Vision Outlooker for Visual Recognition (2021, neutral): “6 Seg-L-Mask/16 [46] ViT-Large ImgNet-22k 53.”
Visual Prompt Tuning (2022, background): “The triumph of the Transformer architecture also extends to various computer vision tasks, including image classification [16,48], object detection [7,45], semantic segmentation [66,83], panoptic segmentation [73], video understanding [21] and fewshot learning [15], surpassing previous state-of-the-”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “More recent works [26, 27, 44, 56] have shown the success of transformer-based methods in semantic segmentation following its success in language and vision [3, 49].”
SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “Recently, ViT-based approaches [17, 24, 44, 46, 50] have shown competitive performance on segmentation tasks compared to ConvNets.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “Follow-up methods find context to play an important role for precise per-pixel classification and focus on designing customized context modules [7,8,63] or self-attention variants [21,26,45,55,61,64].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Transformers in Vision: A Survey (2021, methodology): “Below, we discuss axial self-attention [206], a cross-modal approach [232] that can segment regions corresponding to a given language expression, and ViTs based segmentation architectures [181, 222, 251].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, neutral): “5 ViT-L/16‡ [23] Segmenter [53] 334M - 51.”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, methodology): “Concurrent with our work, Segmenter [78] and MaskFormer [20] formulated an end-to-end strategy from a mask classification perspective, same as MaX-DeepLab [83], but extends from panoptic segmentation to semantic segmentation.”
Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model (2024, neutral): “Vim-Ti DeiT-Ti”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Lately, Transformer based solutions [20, 100, 118, 139] attained growing attention; enjoying the flexibility in long-range dependency modeling, fully attentive solutions yield impressive results.”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Visual Transformers (VTs) are progressively emerging architectures in computer vision as an alternative to standard Convolutional Neural Networks (CNNs), and they have already been applied to many tasks, such as image classification [17, 53, 61, 35, 58, 60, 33, 59], object detection [4, 66, 14], seg”
LMSeg: Language-guided Multi-dataset Segmentation (2023, methodology): “More recently, SETR (Zheng et al., 2021) and Segmenter (Strudel et al., 2021) replace the traditional convolutional backbone with Vision Transformers (ViT) (Dosovitskiy et al., 2021) to capture long-range context from the ﬁrst layer.”
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation (2022, neutral): “This concept was further improved in different aspects including increasing the receptive field while preserving spatial details [2, 3, 6, 48, 65, 74, 83], integrating context information [30,75,76,78,91], attention mechanisms [18,32,69,84], refining boundaries [15,36,77], and Transformer-based arch”
VLP: A Survey on Vision-language Pre-training (2022, background): “Optical character recognition (OCR).[99, 100] OCR generally refers to extracting handwritten or printed text from images (such as street signs and photos of products) as well as documents (articles, bills, invoices, financial reports, etc.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “Previous work follows two principles: 1) Directly considering semantic segmentation as per-pixel classification task [29, 39, 45, 52, 55]; 2) Decoupling the mask generation and the semantic classification [8, 9, 51].”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, background): “However, most of these works only address a single modality combination.”