UPSNet: A Unified Panoptic Segmentation Network

By Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun. 2019

465 citations recorded by the source corpus.

In this paper, we propose a unified panoptic segmentation network (UPSNet) for tackling the newly proposed panoptic segmentation task. On top of a single backbone residual network, we first design a deformable convolution based semantic segmentation head and a Mask R-CNN style instance segmentation head which solve these two subtasks simultaneously. More importantly, we introduce a parameter-free panoptic head which solves the panoptic segmentation via pixel-wise classification. It first leverages the logits from the previous two heads and then innovatively expands the representation for enabling prediction of an extra unknown class which helps better resolving the conflicts between semantic and instance segmentation. Besides, it handles the challenge caused by the varying number of instances and permits back propagation to the bottom modules in an end-to-end manner. Extensive experimental results on Cityscapes, COCO and our internal dataset demonstrate that our UPSNet achieves state-of-the-art performance with much faster inference. Code has been made available at: https://github.com/uber-research/UPSNet

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “UPSNet [85] develops an eﬃcient parameter-free panoptic head for fusing ‘thing’ and ‘stuﬀ’, which is further explored by Li et al .”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “Architectures for panoptic segmentation either combine the best of specialized semantic and instance segmentation architectures into a single framework [11,27,31,60] or design novel objectives that equally treat semantic regions and instance objects [5, 52].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Each level of this proxy tree involves manually-designed modules, such as anchors [77], box assignment rules [105], non-maximum suppression (NMS) [7], thing-stuff merging [98], etc.”
Object-Contextual Representations for Semantic Segmentation (2019, neutral): “In general, the performance of “Panoptic-FPN + OCR” is very competitive compared to various recent methods [66,44,69].”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “UPSNet [76] develops a parameter-free panoptic head which resolves the conflicts in ‘thing’-‘stuff’ fusion by predicting an extra unknown class.”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “In this work, we modify the design in [26] by inserting a Group Normalisation operation [25] after each convolution, which has been observed to help stabilise training.”
AdaptIS: Adaptive Instance Selection Network (2019, neutral): “4 UPSNet [27] ImageNet+COCO ResNet-101 60.”
Panoptic SegFormer (2021, methodology): “Since then, models such as Panoptic FPN [2], UPSNet [4] and AUNet [23] have improved the accuracy and reduced the computational overhead by combining instance segmentation and semantic segmentation into a single model.”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, methodology): “As the semantic segmentation model does not distinguish instances, we further adopt a state-of-the-art instance segmentation method [108] to obtain instance hypotheses for “car”, “truck”, and “pedestrian”.”
Learning Instance Occlusion for Panoptic Segmentation (2019, methodology): “For ResNet-50 backbone, we additionally add one experiment that adopts the design from [38] which uses 2 layers of deformable convolution followed by a bilinear upsampling to the 1/4 scale.”
Video Panoptic Segmentation (2020, methodology): “Since then, much research [7, 8, 10, 15, 17–20, 24, 31, 37, 40] has been actively gathered to propose new approaches to this uniﬁed task, which is now a de facto standard of visual recognition task.”
Bidirectional Graph Reasoning Network for Panoptic Segmentation (2020, methodology): “Panoptic-FPN [19] 39.0 45.9 28.7 Panoptic-FPN-D † 39.9 46.9 29.3 AUNet [22] 39.6 49.1 25.2 OANet [26] 39.0 48.3 26.6 UPSNet-C [33] 41.5 47.5 32.6 UPSNet-CP [33] 41.5 47.3 32.8 UPSNet [33] 42.5 48.5 33.4 SpatialFlow [3] 40.9 46.8 31.9 Our BGRNet 43.2 49.8 33.4 attention [30] and we applied 3…”