Panoptic Segmentation

By Alexander Kirillov, Kaiming He, Ross B. Girshick, Carsten Rother, Piotr Doll'ar. 2018

1,744 citations recorded by the source corpus.

We propose and study a task we name panoptic segmentation (PS). Panoptic segmentation unifies the typically distinct tasks of semantic segmentation (assign a class label to each pixel) and instance segmentation (detect and segment each object instance). The proposed task requires generating a coherent scene segmentation that is rich and complete, an important step toward real-world vision systems. While early work in computer vision addressed related image/scene parsing tasks, these are not currently popular, possibly due to lack of appropriate metrics or associated recognition challenges. To address this, we propose a novel panoptic quality (PQ) metric that captures performance for all classes (stuff and things) in an interpretable and unified manner. Using the proposed metric, we perform a rigorous study of both human and machine performance for PS on three existing datasets, revealing interesting insights about the task. The aim of our work is to revive the interest of the community in a more unified view of image segmentation. For more analysis and up-to-date results, please check the arXiv version of the paper: {\small\url{https://arxiv.org/abs/1801.00868}}.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Object Detection with Transformers (2020, methodology): “This procedure guarantees that the final masks have no overlaps and, therefore, DETR does not require a heuristic [19] that is often used to align different masks.”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “We then convert the ImageNet pretrained Axial-ResNet to Axial-DeepLab, and report results on COCO [59], Mapillary Vistas [65], and Cityscapes [23] for panoptic segmentation, evaluated by panoptic quality (PQ) [48].”
UPSNet: A Unified Panoptic Segmentation Network (2019, methodology): “Recall that, the panoptic quality (PQ) [17] metric for panoptic segmentation is deﬁned as, which consist of two parts: recognition quality (RQ) and semantic quality (SQ).”
Panoptic Feature Pyramid Networks (2019, methodology): “ch norm (BN) [28]. When used in ﬁne-tuning, we replace BN with a ﬁxed channel-wise afﬁne transformation, as is typical [25]. 3.2. Inference and Training Panoptic inference: The panoptic output format [30] requires each output pixel to be assigned a single class label (or void) and instance id (the i”
Segment Anything (2023, background): “Segmentation is a broad ﬁeld: there’s interactive segmentation [57, 109], edge detection [3], super pixelization [85], object proposal generation [2], foreground segmentation [94], semantic segmentation [90], instance segmentation [66], panoptic segmentation [59], etc .”
A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, background): “Generic segmentation mainly includes instance, semantic and panoptic segmentation [3, 15, 21], with respect to different semantics.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “With the capability of per-pixel and instance-level understanding, a natural step was taken to formulate panoptic segmentation [12,35,68].”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, background): “Since deep learning, end-to-end approaches proved their success on problems such as semantic segmentation [37,39], instance segmentation [22], panoptic segmentation [29, 52].”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “Here, we provide more results of Mask2Former with different backbones on COCO panoptic [28] for panoptic segmentation, COCO [35] for instance segmentation and ADE20K [65] for semantic segmentation.”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “Our work is of course relevant to some image-level recognition tasks, such as semantic segmentation [40,8, 9], instance segmentation [17,21,28], panoptic segmentation [27,62,32], large vocabulary instance segmentation [16, 60], etc.”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “To verify the generalization ability of our method, we apply OCR scheme on the more challenging panoptic segmentation task [30], which unifies both the instance segmentation task and the semantic segmentation task.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “Our bottom-up Panoptic-DeepLab is conceptually simple and delivers state-of-the-art panoptic segmentation re-sults [7].”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “This situation is particularly embarrassing in front of the booming “mid-level” vision techniques nowadays: we can already detect and segment almost everything in an image [12, 17, 36].”
Learning to Fuse Things and Stuff (2018, methodology): “Their work focuses on the flexibility to handle weak supervision, at the expense of accuracy, yielding significantly worse performance than the panoptic baseline of [15], even when fully supervised.”