Unifying Training and Inference for Panoptic Segmentation

By Qizhu Li, Xiaojuan Qi, Philip H. S. Torr. 2020

70 citations recorded by the source corpus.

We present an end-to-end network to bridge the gap between training and inference pipeline for panoptic segmentation, a task that seeks to partition an image into semantic regions for "stuff" and object instances for "things". In contrast to recent works, our network exploits a parametrised, yet lightweight panoptic segmentation submodule, powered by an end-to-end learnt dense instance affinity, to capture the probability that any pair of pixels belong to the same instance. This panoptic submodule gives rise to a novel propagation mechanism for panoptic logits and enables the network to output a coherent panoptic segmentation map for both "stuff" and "thing" classes, without any post-processing. Reaping the benefits of end-to-end training, our full system sets new records on the popular street scene dataset, Cityscapes, achieving 61.4 PQ with a ResNet-50 backbone using only the fine annotations. On the challenging COCO dataset, our ResNet-50-based network also delivers state-of-the-art accuracy of 43.4 PQ. Moreover, our network flexibly works with and without object mask cues, performing competitively under both settings, which is of interest for applications with computation budgets.

Subfields: Computer Vision and Pattern Recognition, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Ocean Engineering

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “[49] for end-to-end training of panoptic segmentation models.”
K-Net: Towards Unified Image Segmentation (2021, methodology): “Mainstream frameworks add a semantic segmentation branch [28, 31, 55, 59] on an instance segmentation framework or adopt different pixel grouping strategies [11, 61] based on a semantic segmentation method.”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “Our work is of course relevant to some image-level recognition tasks, such as semantic segmentation [40,8, 9], instance segmentation [17,21,28], panoptic segmentation [27,62,32], large vocabulary instance segmentation [16, 60], etc.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “For example, UPSNet [98] developed a parameter-free panoptic head, which facilitates unified training and inference [55].”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, methodology): “[51] DCN-101 [17] 47.”
Fully Convolutional Networks for Panoptic Segmentation (2020, methodology): “Foreground things are usually separated and represented with boxes [18, 52, 5, 24] or aggregated according to center offsets [51], while background stuff is often predicted with a parallel FCN [33] branch.”
Fully Convolutional Networks for Panoptic Segmentation With Point-Based Supervision (2021, methodology): “Even compared with the leading box-based models [33], [34], the proposed method still achieves comparable result.”
Towards Bounding-Box Free Panoptic Segmentation (2020, background): “In [21] the authors make up for the reduced accuracy by using an affinity network but this is at the cost of computational complexity.”