DeeperLab: Single-Shot Image Parser

By Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen. 2019

197 citations recorded by the source corpus.

We present a single-shot, bottom-up approach for whole image parsing. Whole image parsing, also known as Panoptic Segmentation, generalizes the tasks of semantic segmentation for 'stuff' classes and instance segmentation for 'thing' classes, assigning both semantic and instance labels to every pixel in an image. Recent approaches to whole image parsing typically employ separate standalone modules for the constituent semantic and instance segmentation tasks and require multiple passes of inference. Instead, the proposed DeeperLab image parser performs whole image parsing with a significantly simpler, fully convolutional approach that jointly addresses the semantic and instance segmentation tasks in a single-shot manner, resulting in a streamlined system that better lends itself to fast processing. For quantitative evaluation, we use both the instance-based Panoptic Quality (PQ) metric and the proposed region-based Parsing Covering (PC) metric, which better captures the image parsing quality on 'stuff' classes and larger object instances. We report experimental results on the challenging Mapillary Vistas dataset, in which our single model achieves 31.95% (val) / 31.6% PQ (test) and 55.26% PC (val) with 3 frames per second (fps) on GPU or near real-time speed (22.6 fps on GPU) with reduced accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “DeeperLab [87] predicts bounding box four corners and object centers for class-agnostic instance segmentation.”
Panoptic Segmentation (2018, neutral): “[23, 47, 48, 27, 22, 21, 17] address PS).”
K-Net: Towards Unified Image Segmentation (2021, methodology): “Mainstream frameworks add a semantic segmentation branch [28, 31, 56, 61] on an instance segmentation framework or adopt different pixel grouping strategies [11, 63] based on a semantic segmentation method.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Then, instance segments are obtained by grouping ‘thing’ pixels with various methods, such as instance center regression [44, 86, 70, 100, 20], Watershed transform [88, 3, 8], Hough-voting [4, 53, 8], or pixel affinity [45, 66, 81, 30, 8].”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “Panoptic segmentation: Given the predicted semantic segmentation and class-agnostic instance segmentation re-sults, we adopt a fast and parallelizable method to merge the results, following the “majority vote” principle proposed in DeeperLab [16].”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, methodology): “Method mIOU Seamless [46] 50.4 DeeperLab [47] 55.3 Panoptic DeepLab [23] 56.8 Panoptic DeepLab ( Ensemble ) [23] 58.7 Ours 61.1”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, methodology): “We study the effects of on-line hard example (or pixel) mining (OHEM) [4, 61, 65] in Table 8.”
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid (2019, methodology): “To the best of our knowledge, DeeperLab [48] is currently the only proposal-free method to report COCO result.”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “…at tackling panoptic segmentation often resort to two separate networks for instance and semantic segmentation, recent works [18, 16, 13, 26, 27] are able to improve the overall efﬁciency by constructing the two branches on a single, shared feature extractor, and training the multi-head,…”
AdaptIS: Adaptive Instance Selection Network (2019, methodology): “0 DeeperLab [28] ImageNet Xception-71 56.”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, methodology): “Other proxy-based methods typically start with semantic segments [11,13,16] and group ‘thing’ pixels into instance segments with various proxy tasks, such as instance center regression [19,42,56,67,79,83,90], Watershed transform [4, 8, 81], Hough-voting [5, 8, 51], or pixel affinity [8, 29, 43, 61, ”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, methodology): “izer without weight decay are adopted. On Cityscapes, we employ training crop size equal to 1025 2049 with batch size 32, and 180K training iterations. Similar to other works on panoptic segmentation [38,43,37,85,61,87,78], we re-assign to void label all ‘stu‘ segments whose areas are smaller than a”
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers (2021, background): “The panoptic segmentation literature mainly treats this problem as a joint task of instance segmentation and semantic segmentation where things and stuff are handled separately [18, 19].”
FastDepth: Fast Monocular Depth Estimation on Embedded Systems (2019, methodology): “Skip connections have been previously been used in networks for image segmentation such as U-Net [33] and DeeperLab [34], showing that they can be beneﬁcial in networks producing dense outputs.”
Panoptic SegFormer (2021, result): “Several works further consider bottom-up (proposal-free) instance segmentation approaches but still maintain similar separate strategies [6–10].”
SOGNet: Scene Overlap Graph Network for Panoptic Segmentation (2019, background): “Follow-up studies train semantic and instance segmentation in an endto-end network by sharing backbone (de Geus, Meletis, and Dubbelman 2018; Kirillov et al. 2019a; Li et al. 2019b; Liu et al. 2019; Xiong et al. 2019; Porzi et al. 2019; Yang et al. 2019; Li et al. 2018).”
Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation (2020, unknown): “Recent studies on panoptic segmentation mainly fall into three categories: combining methods from separate models [20], simple multi-branch schemes [11, 42], and inter-modular fusion methods [23, 40, 38].”
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation (2019, methodology): “(2) Dilated convolution [44, 5, 6, 7, 8, 4, 28, 43, 11, 12] ( a.k.a. “atrous” convolution) is used to remove several stride convolutions/max poolings to preserve feature map resolution.”
Video Panoptic Segmentation (2020, background): “…the class-agnostic non-maximum suppression with the box IoU [20] ResNet-101 59.0 54.8 62.1 PanopticFPN [15] ResNet-101 58.1 52.0 62.5 DeeperLab [40] Xception-71 56.5 - - Seamless [31] ResNet-50 59.8 54.6 63.6 AdaptIS [34] ResNet-50 59.0 55.8 61.3 TASCNet [18] ResNet-50 55.9 50.6 59.8 UPSNet…”