Video Instance Segmentation

By Linjie Yang, Yuchen Fan, Ning Xu. 2019

636 citations recorded by the source corpus.

In this paper we present a new computer vision task, named video instance segmentation. The goal of this new task is simultaneous detection, segmentation and tracking of instances in videos. In words, it is the first time that the image instance segmentation problem is extended to the video domain. To facilitate research on this new task, we propose a large-scale benchmark called YouTube-VIS, which consists of 2,883 high-resolution YouTube videos, a 40-category label set and 131k high-quality instance masks. In addition, we propose a novel algorithm called MaskTrack R-CNN for this task. Our new method introduces a new tracking branch to Mask R-CNN to jointly perform the detection, segmentation and tracking tasks simultaneously. Finally, we evaluate the proposed method and several strong baselines on our new dataset. Experimental results clearly demonstrate the advantages of the proposed algorithm and reveal insight for future improvement. We believe the video instance segmentation task will motivate the community along the line of research for video understanding.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Instance Segmentation for Autonomous Log Grasping in Forestry Operations (2022, background): “approaches performing detection directly from video streams [35] could also be envisioned.”
Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “All frames are resized to 640 × 360 during both training and inference for fair comparisons with previous works (Yang et al., 2019; Cao et al., 2020;Athar et al., 2020).”
Transformers in Vision: A Survey (2021, neutral): “The overall results are competitive among the single model approaches on YouTube VIS dataset [162], but performs somewhat lower compared to more complex CNN-based models such as MaskProp [163].”
End-to-End Video Instance Segmentation with Transformers (2020, methodology): “Video instance segmentation (VIS), recently proposed in [30], requires one to simultaneously classify, segment and track object instances of interest in a video sequence.”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, methodology): “Unlike YouTube-VIS, we do not have a predeﬁned taxonomy for objects.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, neutral): “Figure 1: We compare our video instance segmentation results with those produced by MaskTrack R-CNN [5].”
Language as Queries for Referring Video Object Segmentation (2022, positive): “Beyond image ﬁeld, VisTR [50] extends the framework for video instance segmentation (VIS) [55] task and solves the problem in a direct end-to-end parallel sequence decoding manner.”
Video Instance Segmentation 2019: A Winning Approach for Combined Detection, Segmentation, Classification and Tracking. (2019, methodology): “To adapt this network to VIS, we created a training set by combining the YT-VIS [28], COCO [10] and OpenIm-ages [8] datasets.”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “Considering reference-query matching, we classify methods as two-stage [43, 38, 42] and one-stage [14, 35, 34, 36, 20, 10, 29, 28, 7] basing on whether matching with proposed region-of-interest.”
Crossover Learning for Fast Online Video Instance Segmentation (2021, methodology): “We evaluate the proposed CrossVIS on three challenging video instance segmentation benchmarks, i.e ., YouTube-VIS-2019 [72], OVIS [51] and YouTube-VIS-2021 [71].”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “And like image segmentation, most existing methods are specialist models comprising multiple stages with pipe-lined frameworks, e.g., track-detect [67, 57, 36, 6, 44], clip-match [3, 5], propose-reduce [37].”
Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, methodology): “For example, MaskTrack R-CNN [43] predicts an extra embedding vector for each instance and use an external memory to store them for tracking across frames.”
HOTA: A Higher Order Metric for Evaluating Multi-object Tracking (2020, methodology): “Commonly used representations include 2D bounding boxes [39,28], 3D bounding boxes [73,14], segmentation masks [74,83], point estimates in 2D or 3D [23], and human pose skeletons [1].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Itcanreﬁnetheexistingvideo instancesegmentationdatasets[50].”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, methodology): “[18] Linjie Yang, Yuchen Fan, and Ning Xu.”
Opening up Open World Tracking (2021, methodology): “Such a factorization is not possible with other metrics such as Track mAP [90] and IDF1 [68].”
Instances as Queries (2021, methodology): “Following the standard setting in [59], we first pre-train the instance segmentation model on COCO train2017, then we finetune the corresponding VIS model on YouTube-VIS train set for 12 epochs.”
Associating Objects with Transformers for Video Object Segmentation (2021, background): “As to related areas of VOS, the simple yet effective identification mechanism should also be promising for many tasks requiring multi-object matching, such as interactive video object segmentation [14, 17], video instance segmentation [1, 26, 31], and multi-object tracking [15, 27, 30].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “For example, in [82], [210], [253], Mask R-CNN [180] is adapted for VIS/MOTS by adding a tracking branch for cross-frame instance association.”
Track to Detect and Segment: An Online Multi-Object Tracker (2021, methodology): “We observe that TraDeS outperforms the baseline tracker by a large margin on both nuScenes and YouTube-VIS.”