Per-Pixel Classification is Not All You Need for Semantic Segmentation

By Bowen Cheng, Alexander Gerhard Schwing, Alexander Kirillov. 2021

2,101 citations recorded by the source corpus.

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Alan L. Yuille
3. Quoc V. Le
4. Luc Van Gool
5. Dacheng Tao
6. Trevor Darrell
7. Mohit Bansal
8. Jiashi Feng

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Ming-Hsuan Yang
5. Alan L. Yuille
6. Trevor Darrell
7. Jitendra Malik
8. Andrew Zisserman

Citations in the corpus

• Segment Anything (2023, methodology): “We build on Transformer vision models [14, 33, 20, 62] with specific tradeoffs for (amortized) real-time performance.”
• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “We collect foundation models from Hug-gingFace Transformers [43] 2 , Maskformer [10] 3 and Con-trolNet [53] 4 .”
• Instance Segmentation for Autonomous Log Grasping in Forestry Operations (2022, methodology): “Also based on DETR, MaskFormer [20] first segments the input image into a fixed number of masks and then classifies them.”
• Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “MaskFormer [14] shows that mask classification based on DETR not only performs well on panoptic segmentation but also achieves state-of-the-art on semantic segmentation.”
• K-Net: Towards Unified Image Segmentation (2021, methodology): “In contrast to K-Net that directly uses learned kernels to predict masks and progressively refines the masks and kernels, MaX-DeepLab [52] and MaskFormer [12] rely on queries and Transformer [51] to produce dynamic kernels for the final mask prediction.”
• ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “Transformer have also been explored in object detection [10,82,16,63], semantic segmentation [65,67,14], and video representation learning [2,45,22].”
• Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation (2022, positive): “Meanwhile, MaskFormer [9] regards the semantic segmentation as a mask classification task and introduces the transformer [37] to obtain a better performance.”
• Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, methodology): “Following the common practices [7, 8], we add the absolute position embedding in each transformer block for both visual tokens and query tokens.”
• Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, methodology): “For fair comparisons, we do not consider complex segmentation heads such as MaskFormer [11].”
• CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation (2023, background): “Mask2Former [7] and MaskFormer [8] propose universal transformer-based architectures for semantic, instance and panoptic segmentation.”
• ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “Local embeddings: We employ a universal instance segmentation model [56, 57]2 to produce a set of R class-agnostic”
• VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, background): “Researchers have significantly promoted image segmentation performance with a variety of deep learning techniques, such as convolutional neural network [34, 35], vision transformers [36, 37], prototype learning [38, 39], etc.”
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “Built upon Mask2Former [Cheng et al., 2021], we employ the video encoder of InternVideo2 as backbone with ViT-adapter [Chen et al., 2022b] for features.”
• ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “As we described in Subsec 3.4, we propose that applying non-mutually exclusive objective functions like using binary cross entropy (BCE) with sigmoid performs better on zero-shot semantic segmentation.”
• CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, methodology): “method epochs memory params FLOPs FPS PQ MaskFormer-SwinB [2] 300 - 102M 411G 8.”
• UViM: A Unified Modeling Approach for Vision with Learned Guiding Codes (2022, methodology): “To address this shortcoming, standard approaches turn to using additional modeling components such as, for example, anchor boxes [36, 29], non-maximal suppression [36, 29], matching losses [2, 5, 6] or conditional random fields [3, 54].”