Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network

By Daan de Geus, Panagiotis Meletis, Gijs Dubbelman. 2018

80 citations recorded by the source corpus.

We present a single network method for panoptic segmentation. This method combines the predictions from a jointly trained semantic and instance segmentation network using heuristics. Joint training is the first step towards an end-to-end panoptic segmentation network and is faster and more memory efficient than training and predicting with two networks, as done in previous work. The architecture consists of a ResNet-50 feature extractor shared by the semantic segmentation and instance segmentation branch. For instance segmentation, a Mask R-CNN type of architecture is used, while the semantic segmentation branch is augmented with a Pyramid Pooling Module. Results for this method are submitted to the COCO and Mapillary Joint Recognition Challenge 2018. Our approach achieves a PQ score of 17.6 on the Mapillary Vistas validation set and 27.2 on the COCO test-dev set.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

UPSNet: A Unified Panoptic Segmentation Network (2019, methodology): “JSIS-Net uses a ResNet-50 wheres RN50-MR-CNN uses two separate ResNet-50-FPNs as the backbone.”
Panoptic Feature Pyramid Networks (2019, methodology): “Results. 7 Figure 6: More Panoptic FPN results on COCO (top) and Cityscapes (bottom) using a single ResNet-101-FPN network. PQ PQ ThPQ St Artemis 16.9 16.8 17.0 LeChen 26.2 31.0 18.9 MPS-TU Eindhoven [16] 27.2 29.6 23.4 MMAP-seg 32.1 38.9 22.0 Panoptic FPN 40.9 48.3 29.7 (a) Panoptic Segmentation on”
Attention-Guided Unified Network for Panoptic Segmentation (2018, methodology): “ of panopic segmentation by combining instance and semantic segmentation models. Later, a weakly-supervised method [20] was proposed on top of initialized semantic results, and an end-to-end approach [8] was designed to combine both FG and BG cues. However, their performance is far from the benchmar”
Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation (2020, unknown): “Recent studies on panoptic segmentation mainly fall into three categories: combining methods from separate models [20], simple multi-branch schemes [11, 42], and intermodular fusion methods [23, 40, 38].”
Video Panoptic Segmentation (2020, background): “Since then, a large number of works [7,8,10,15,17–20,24, 31,37,40] have proposed learning-based approaches to this new benchmark task, conﬁrming its importance to the ﬁeld.”
SpatialFlow: Bridging All Tasks for Panoptic Segmentation (2019, methodology): “…well aware of locations of objects is fundamental to many vision tasks, e.g., object detection (Ren et al. 2015; Dai et al. 2016; Lin et al. 2017b), instance segmentation (He et al. 2017; Fu, Shvets, and Berg 2019), semantic segmentation (Chen et al. 2018b; Chen et al. 2017; Zhao et al. 2017).”
An End-To-End Network for Panoptic Segmentation (2019, unknown): “JSIS-Net [10] proposes a single network with the instance segmentation head [14] and the pyramid stuff segmentation head [48], following heuristics to merge two kinds of outputs.”
EfficientPS: Efficient Panoptic Segmentation (2020, methodology): “Most architectures (Kirillov et al, 2019; Porzi et al, 2019; Li et al, 2019b; de Geus et al, 2018) employ a standard post-processing step (Kirillov et al, 2019) that adopts instance-specific ‘thing’ segmentation from the instance head and ‘stuff’ segmentation from the semantic head.”
Bidirectional Graph Reasoning Network for Panoptic Segmentation (2020, methodology): “The panoptic segmentation task is to assign each pixel in an image a semantic label and an instance id. Current methods typically address this issue with a uniﬁed model using two branches for foreground things and background stuff separately [8, 19, 21, 22].”