Fully Convolutional Networks for Panoptic Segmentation

By Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li, Jiaya Jia. 2020

199 citations recorded by the source corpus.

In this paper, we present a conceptually simple, strong, and efficient framework for panoptic segmentation, called Panoptic FCN. Our approach aims to represent and predict foreground things and background stuff in a unified fully convolutional pipeline. In particular, Panoptic FCN encodes each object instance or stuff category into a specific kernel weight with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly. With this approach, instance-aware and semantically consistent prosperties for things and stuff can be respectively satisfied in a simple generate-kernel-then-segment workflow. Without extra boxes for localization or instance separation, the proposed approach outperforms previous box-based and -free models with high efficiency on COCO, Cityscapes, and Mapillary Vistas datasets with single scale input. Our code is made publicly available at https://github.com/Jia-Research-Lab/PanopticFCN.1

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

K-Net: Towards Unified Image Segmentation (2021, methodology): “Recent studies apply dynamic kernels to generate instance [49, 55] or panoptic [34] segmentation predictions directly.”
CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation (2022, methodology): “Other proxy-based methods typically start with semantic segments [11,13,16] and group ‘thing’ pixels into instance segments with various proxy tasks, such as instance center regression [19,42,56,67,79,83,90], Watershed transform [4, 8, 81], Hough-voting [5, 8, 51], or pixel affinity [8, 29, 43, 61, ”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Based on IS subtask, the previous works can also be divided into two categories: top-down approaches [87], [88] and bottom-up approaches [84], [89], according to the way to generate instance masks.”
FaPN: Feature-aligned Pyramid Network for Dense Image Prediction (2021, unknown): “We implement FaPN in modern dense prediction frameworks (Faster R-CNN [41], Mask R-CNN [13], PointRend [21], Mask-Former [8], PanopticFPN [19], and PanopticFCN [23]), and demonstrate its efficacy on object detection, semantic, instance and panoptic segmentation.”
ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning (2024, methodology): “Following this, some methods [27, 43, 48] introduced significant improvements by employing dynamic convolutions in a fully convolutional paradigm.”
k-means Mask Transformer (2022, methodology): “…clustering-based methods usually operate on semantic segments [13,15,18] and group ‘thing’ pixels into instance segments with various representations, such as instance center regression [50,89,77,104,22,94,63], Watershed transform [91,5], Hough-voting [6,59,92], or pixel aﬃnity [51,69,85,36,47].”
Learning to Upsample by Learning to Sample (2023, background): “Dense prediction refers to a branch of tasks that require point-wise label prediction, such as semantic/instance/panoptic segmentation [2, 39, 40, 8, 7, 13, 11, 16, 19], object detection [33, 4, 24, 36], and monocular depth estimation [38, 18, 3, 21].”
LISA: Reasoning Segmentation via Large Language Model (2023, background): “, 2022) and panoptic segmentation (Kirillov et al., 2019; Xiong et al., 2019; Cheng et al., 2020; Li et al., 2021) has introduced various architectural innovations for instance-level segmentation, including DETR (Carion et al.”
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers (2021, methodology): “[21] proposed Panoptic FCN where the panoptic segmentation pipeline is simplified with a “top-down meets bottom-up” two-branch design similar to CondInst [22].”
DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection (2022, methodology): “Further improvement of the CNN-based detectors such as K-Net [56] and Panoptic-FCN [31] further introduce dynamic kernels and replace the static kernels in the convolution layers to improve the ﬂexibility of models.”
Panoptic SegFormer (2021, methodology): “8M 137G Panoptic FCN [13] R50-FPN 36 43.”