Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

By Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen. 2019

690 citations recorded by the source corpus.

In this work, we introduce Panoptic-DeepLab, a simple, strong, and fast system for panoptic segmentation, aiming to establish a solid baseline for bottom-up methods that can achieve comparable performance of two-stage methods while yielding fast inference speed. In particular, Panoptic-DeepLab adopts the dual-ASPP and dual-decoder structures specific to semantic, and instance segmentation, respectively. The semantic segmentation branch is the same as the typical design of any semantic segmentation model (e.g., DeepLab), while the instance segmentation branch is class-agnostic, involving a simple instance center regression. As a result, our single Panoptic-DeepLab simultaneously ranks first at all three Cityscapes benchmarks, setting the new state-of-art of 84.2% mIoU, 39.0% AP, and 65.5% PQ on test set. Additionally, equipped with MobileNetV3, Panoptic-DeepLab runs nearly in real-time with a single 1025x2049 image (15.8 frames per second), while achieving a competitive performance on Cityscapes (54.1 PQ% on test set). On Mapillary Vistas test set, our ensemble of six models attains 42.7% PQ, outperforming the challenge winner in 2018 by a healthy margin of 1.5%. Finally, our Panoptic-DeepLab also performs on par with several top-down approaches on the challenging COCO dataset. For the first time, we demonstrate a bottom-up approach could deliver state-of-the-art results on panoptic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Lastly, following Panoptic-DeepLab [20], we adopt exactly the same stem [81] of three convolutions, dual decoders, and prediction heads.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, neutral): “This effort, however, led to new specialized panoptic architectures [11] instead of unifying the previous tasks (see Fig.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “For panoptic segmentation, Mask2Former with Swin-L backbone outperforms the state-of-the-art Panoptic-DeepLab [11] with SWideR-net [9] using single-scale inference.”
iShape: A First Step Towards Irregular Shape Instance Segmentation (2021, neutral): “Meanwhile, overlapped objects that share the same center point challenge center-based methods[21, 22, 23].”
Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, methodology): “In addition, we use the R101c model [7] which replaces the first 7× 7 convolution layer of R101 with 3 consecutive 3× 3 convolutions and which is popular in the semantic segmentation community [52, 8, 9, 23, 50, 11].”
ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes (2021, background): “Recalling the history of success with other complex segmentation and detection datasets ( e.g. from mIoU 57% in 2015 [4] to 84% in 2020 [15] on CityScapes [17], or from AP50 29 .”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “The state-of-theart method in this regime, Axial-DeepLab [89], along with other box-free methods [100, 21, 11], predicts pixel-wise offsets to pre-defined instance centers.”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, methodology): “Nevertheless, our model is still superior to Axial-DeepLab when multi-scale inference is adopted on Cityscapes validation set.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “It is reported in [26] that a slightly-modiﬁed HRNet combined with ASPP achieved the best performance for Mapillary panoptic segmentation in the single model case.”
Segmenter: Transformer for Semantic Segmentation (2021, neutral): “3 Panoptic-Deeplab [12] Xception-71 81.”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, methodology): “4 higher than the next closest method, Panoptic Deeplab [23], which uses ensemble of models to achieve 58.”
Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “As described in § 4.2.4, we use a semantic segmentation model [10] to detect movable objects.”
Pointly-Supervised Instance Segmentation (2021, background): “Alternatively, bottom-up segmentation methods output masks indirectly, forming them from a set of predicted cues such as instance boundaries [24], energy levels [3], and offset vectors to center points [8].”
Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation (2020, methodology): “precision (AP), and panoptic quality (PQ) to evaluate the semantic, instance, and panoptic segmentation results, respectively. The state-of-art bottom-up panoptic segmentation model, Panoptic-DeepLab [15], is included in our proposed iterative semi-supervised learning pipeline. Panotic-DeepLab is a ”
HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation (2019, methodology): “(2) Dilated convolution [44, 5, 6, 7, 8, 4, 28, 43, 11, 12] ( a.k.a. “atrous” convolution) is used to remove several stride convolutions/max poolings to preserve feature map resolution.”
CoMFormer: Continual Learning in Semantic and Panoptic Segmentation (2022, methodology): “Initially, meth-ods proposed to combine each task-speciﬁc architectures [10,27,32,42] or deﬁned new specialized architectures and objective functions for the panoptic task [1,50,51], drifting from a general solution for all the segmentation tasks.”
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation (2022, positive): “Furthermore, multi-scale input inference, where predictions from scaled versions of an image are combined via average or max pooling, is often used to obtain better results [4, 6, 9, 75].”