Seamless Scene Segmentation

By Lorenzo Porzi, Samuel Rota Bulò, Aleksander Colovic, Peter Kontschieder. 2019

231 citations recorded by the source corpus.

In this work we introduce a novel, CNN-based architecture that can be trained end-to-end to deliver seamless scene segmentation results. Our goal is to predict consistent semantic segmentation and detection results by means of a panoptic output format, going beyond the simple combination of independently trained segmentation and detection models. The proposed architecture takes advantage of a novel segmentation head that seamlessly integrates multi-scale features generated by a Feature Pyramid Network with contextual information conveyed by a light-weight DeepLab-like module. As additional contribution we review the panoptic metric and propose an alternative that overcomes its limitations when evaluating non-instance categories. Our proposed network architecture yields state-of-the-art results on three challenging street-level datasets, i.e. Cityscapes, Indian Driving Dataset and Mapillary Vistas.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “5% better than Seamless [71].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, the instance segments (‘thing’) and semantic segments (‘stuff’) [13] are fused by merging modules [54, 56, 74, 67, 101] to generate panoptic segmentation.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “During evaluation, due to the sensitivity of PQ [15, 8, 12], we re-assign to ‘VOID’ label all ‘stuff’ segments whose areas are smaller than a threshold.”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, methodology): “Method mIOU Seamless [46] 50.4 DeeperLab [47] 55.3 Panoptic DeepLab [23] 56.8 Panoptic DeepLab ( Ensemble ) [23] 58.7 Ours 61.1”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “AP: Replace global average pooling with an average pooling whose kernel size is the same as training crop size divided by the stride of that feature maps [47].”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “Efforts in this direction include Panoptic Feature Pyramid Network (PFPN) [144], attention-guided network for panoptic segmentation [145], seamless scene segmentation [146], panoptic Deeplab [147], unified panoptic segmentation network [148], and efficient panoptic segmentation [149].”
Unifying Training and Inference for Panoptic Segmentation (2020, neutral): “7 – – – – – – – – – – – – – Seamless [20]† 60.”
Bidirectional Graph Reasoning Network for Panoptic Segmentation (2020, methodology): “A mini-deeplab module was also used to capture more contextual information in [29].”
Towards Bounding-Box Free Panoptic Segmentation (2020, methodology): “n networks. Using a MoE approach, the outputs are combined either heuristically or through another sub-network. Although, more recent works propose sharing a common feature backbone for both networks [16,28], this split of tasks restricts the backbone network to the most complex branch. Usually this”
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution (2020, methodology): “…them, our proposed Recursive Feature Pyramid goes through the bottom-up backbone repeatedly to enrich the representation power of FPN. Additionally, we incorporate the Atrous Spatial Pyramid Pooling (ASPP) [13,14] into FPN to enrich features, similar to the mini-DeepLab design in Seamless [55].”
Video Panoptic Segmentation (2020, background): “…suppression with the box IoU [20] ResNet-101 59.0 54.8 62.1 PanopticFPN [15] ResNet-101 58.1 52.0 62.5 DeeperLab [40] Xception-71 56.5 - - Seamless [31] ResNet-50 59.8 54.6 63.6 AdaptIS [34] ResNet-50 59.0 55.8 61.3 TASCNet [18] ResNet-50 55.9 50.6 59.8 UPSNet [37] ResNet-50 59.3 54.6 62.7…”
Fully Convolutional Networks for Panoptic Segmentation (2020, methodology): “Specifically, Panoptic FCN surpasses the leading box-based [36] and box-free [6] models with 0.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, unknown): “Recently, panoptic segmentation [47] is proposed to unify pixel-level and instance-level semantic segmentation, and it starts to get more attentions for autonomous driving [48]–[50].”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, methodology): “odels are about 0.1%). Transfer learning from Cityscapes to Mapillary Vistas: The transfer learning from the large-scale Mapillary Vistas to Cityscapes has been shown to be eective in the literature [95,61,15] and in our work as well, since both datasets contain street-view images. In Tab.6, we expe”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, methodology): “Top-down approaches [43, 67, 50, 54, 60, 94, 51, 10, 47, 91] typically pair Mask R-CNN [27] with a light-weight ‘stuff‘ segmentation branch, while bottom-up approaches [95, 20, 87, 12, 88] group ‘thing‘ pixels from semantic segmentation predictions.”
EfficientPS: Efficient Panoptic Segmentation (2020, methodology): “…(Li et al, 2018b), TASCNet (Li et al, 2018a), Panoptic FPN (Kirillov et al, 2019), AUNet (Li et al, 2019b), UPSNet (Xiong et al, 2019), DeeperLab (Yang et al, 2019), Seamless (Porzi et al, 2019), SSAP (Gao et al, 2019), AdaptIS (Sofiiuk et al, 2019), and Panoptic-DeepLab (Cheng et al, 2019).”
SOGNet: Scene Overlap Graph Network for Panoptic Segmentation (2019, methodology): “Similar to (Kirillov et al. 2019a; Li et al. 2019b; Liu et al. 2019; Xiong et al. 2019; Porzi et al. 2019; Li et al. 2018), we also use ResNets (He et al. 2016) with feature pyramid network (FPN) (Lin et al. 2017) as the shared backbone for our semantic and instance segmentation branches.”