Panoptic Feature Pyramid Networks

By Alexander Kirillov, Ross B. Girshick, Kaiming He, Piotr Doll'ar. 2019

1,521 citations recorded by the source corpus.

The recently introduced panoptic segmentation task has renewed our community's interest in unifying the tasks of instance segmentation (for thing classes) and semantic segmentation (for stuff classes). However, current state-of-the-art methods for this joint task use separate and dissimilar networks for instance and semantic segmentation, without performing any shared computation. In this work, we aim to unify these methods at the architectural level, designing a single network for both tasks. Our approach is to endow Mask R-CNN, a popular instance segmentation method, with a semantic segmentation branch using a shared Feature Pyramid Network (FPN) backbone. Surprisingly, this simple baseline not only remains effective for instance segmentation, but also yields a lightweight, top-performing method for semantic segmentation. In this work, we perform a detailed study of this minimally extended version of Mask R-CNN with FPN, which we refer to as Panoptic FPN, and show it is a robust and accurate baseline for both tasks. Given its effectiveness and conceptual simplicity, we hope our method can serve as a strong baseline and aid future research in panoptic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

End-to-End Object Detection with Transformers (2020, methodology): “We perform experiments on COCO 2017 detection and panoptic segmentation datasets [24,18], containing 118k training images and 5k validation images.”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, methodology): “For example, ResNet34 using visual transformers can outperform ResNet101 while using 2.5x fewer MACs, visual transformer-based FPN uses 6.5x fewer MACs than regular FPNs while achieving similar accuracy.”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “0 Panoptic-FPN [47] ResNet-101 40.”
Panoptic Segmentation (2018, neutral): “[23, 48, 49, 27, 22, 21, 17] address PS).”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, methodology): “One of the earliest architectures in this scope was Panoptic-FPN [22], which introduced separate instance and semantic task branches.”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “Architectures for panoptic segmentation either combine the best of specialized semantic and instance segmentation architectures into a single framework [11, 27, 31, 60] or design novel objectives that equally treat semantic regions and instance objects [5, 52].”
K-Net: Towards Unified Image Segmentation (2021, methodology): “With ResNet-101-FPN backbone, K-Net surpasses Cascade Mask R-CNN with 100 and 256 instance kernels in both accuracy and speed by 0.2 AP and 6.7 FPS, and 0.7 AP and 6 FPS, respectively.”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “As shown in Table 4, when using Semantic FPN [31] for semantic segmentation, PVT-based models consistently outperforms the models based on ResNet [21] or ResNeXt [72].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “Panoptic-FPN [47] 7 3 7 7 7 UPSNet [98] 7 3 7 3 7”
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, methodology): “Baselines We adopt dilated FCN [36] and Semantic FPN [27] as baselines with their results taken from [39].”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “The performance of Panoptic-FPN [29] is reproduced based on the official open-source Detectron2 [65] and we use the 3× learning rate schedule by default.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “[6] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár.”
PointRend: Image Segmentation As Rendering (2019, methodology): “Here, we demonstrate that PointRend can beneﬁt two semantic segmentation models: DeeplabV3 [4], which uses dilated convolutions to make prediction on a denser grid, and Se-manticFPN [21], a simple encoder-decoder architecture.”
Meta R-CNN: Towards General Solver for Instance-Level Low-Shot Learning (2019, methodology): “n low-shot object detection/ segmentation. It is worth noting that, as Meta R-CNN solely remodels the predictor branches into a meta-learner, it potentially can be extended to a broad range of models [15,20,24,46] in the entire R-CNN family. To this Meta R-CNN might enable visual structure predictio”
Attention-Guided Unified Network for Panoptic Segmentation (2018, methodology): “ AUNet achieves the top performance in an endto-end framework. Furthermore, we also establish the bond between proposal-based instance and FCN based semantic segmentation. Most recently works include [22,29,40]. Instance Segmentation: Instance segmentation aims at discriminating different instances ”
PVT v2: Improved baselines with Pyramid Vision Transformer (2021, methodology): “For a fair comparison, we test the performance of PVT v2 backbones by applying it to Semantic FPN [18].”
XCiT: Cross-Covariance Image Transformers (2021, methodology): “We integrate our backbones in two segmentation methods: Semantic FPN [38] and UperNet [74].”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “Efforts in this direction include Panoptic Feature Pyramid Network (PFPN) [144], attention-guided network for panoptic segmentation [145], seamless scene segmentation [146], panoptic Deeplab [147], unified panoptic segmentation network [148], and efficient panoptic segmentation [149].”
Unifying Training and Inference for Panoptic Segmentation (2020, methodology): “In our experiments, we adopt a simple ResNet-FPN backbone that outputs four multi-scale feature maps [19], following a common practice in prior works [13, 26].”
AdaptIS: Adaptive Instance Selection Network (2019, background): “6 Panoptic FPN [14] ImageNet ResNet-101 58.”