Video Panoptic Segmentation

By Dahun Kim, Sanghyun Woo, Joon-Young Lee, In-So Kweon. 2020

205 citations recorded by the source corpus.

Panoptic segmentation has become a new standard of visual recognition task by unifying previous semantic segmentation and instance segmentation tasks in concert. In this paper, we propose and explore a new video extension of this task, called video panoptic segmentation. The task requires generating consistent panoptic segmentation as well as an association of instance ids across video frames. To invigorate research on this new task, we present two types of video panoptic datasets. The first is a re-organization of the synthetic VIPER dataset into the video panoptic format to exploit its large-scale pixel annotations. The second is a temporal extension on the Cityscapes val. set, by providing new video panoptic annotations (Cityscapes-VPS). Moreover, we propose a novel video panoptic segmentation network (VPSNet) which jointly predicts object classes, bounding boxes, masks, instance id tracking, and semantic segmentation in video frames. To provide appropriate metrics for this task, we propose a video panoptic quality (VPQ) metric and evaluate our method and several other baselines. Experimental results demonstrate the effectiveness of the presented two datasets. We achieve state-of-the-art results in image PQ on Cityscapes and also in VPQ on Cityscapes-VPS and VIPER datasets.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “[26] define a video extension of panoptic segmentation [27], which requires generating consistent panoptic segmentation, and in the meantime, associating instances across frames.”
Opening up Open World Tracking (2021, background): “Recent efforts move towards pixel-precise segmentation of tracked objects in video [36, 48, 78, 82, 90] or LiDAR sequences [3], and study performance in the long tail of object classes [16].”
ResNet or DenseNet? Introducing Dense Shortcuts to ResNet (2020, background): “Deep neural networks (DNNs) have achieved state-ofthe-art performance in numerous computer vision tasks [17, 38, 15, 31, 59, 58, 13, 24, 35, 25, 26], and the interpretation of DNNs has also be investigated from the lens of visualization [55, 33, 34] as well as robustness [47, 16, 5, 56, 57, 6, 4, 7]”
Transformer-Based Visual Segmentation: A Survey (2023, background): “Most VIS approaches [52], [93], [94], [95] focus on learning instance-wised spatial, temporal relation, and featurefusion.”
ST-P3: End-to-end Vision-based Autonomous Driving via Spatial-Temporal Feature Learning (2022, methodology): “We evaluate our model by the metric of IoU, existing panoptic quality (PQ), recognition quality (RQ), and segmentation quality (SQ) following metrics in video prediction area [29].”
SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, methodology): “Majority of VIS methods [2, 8, 11, 16, 19, 25, 26] follow the two-stage paradigm based on Mask R-CNN.”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, methodology): “[19] extends the image-level panoptic segmentation [20] to the video domain, which requires generating consistent panoptic segmentation, and in the meantime, associating instances across frames.”
FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras (2021, methodology): “We use the Video Panoptic Quality (VQP) [26] metric deﬁned as: with TP t the set of true positives at timestep t (correctly detected ground truth instances), FP t the set of false positives at timestep t (predicted instances that do not match any ground truth instance), and FN t the set of false neg”
iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering (2020, background): “As such, they can be easily adapted for other video-based vision tasks such as scene understanding [16], panoptic segmentation [21], etc.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, background): “Video panoptic segmentation (VPS) [28] is a direct extension of image panoptic segmentation to the video domain.”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, methodology): “ing scenes have been released in the literature [8], [9], [13], [18], [25], [34], [36], [45], [52], [56], [62], [65], [70], [80], [84],”