Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation

By Gedas Bertasius, Lorenzo Torresani. 2019

197 citations recorded by the source corpus.

We introduce a method for simultaneously classifying, segmenting and tracking object instances in a video sequence. Our method, named MaskProp, adapts the popular Mask R-CNN to video by adding a mask propagation branch that propagates frame-level object instance masks from each video frame to all the other frames in a video clip. This allows our system to predict clip-level instance tracks with respect to the object instances segmented in the middle frame of the clip. Clip-level instance tracks generated densely for each frame in the sequence are finally aggregated to produce video-level object instance segmentation and classification. Our experiments demonstrate that our clip-level instance segmentation makes our approach robust to motion blur and object occlusions in video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset, outperforming the ICCV 2019 video instance segmentation challenge winner despite being much simpler and using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes). The project page is at: https://gberta.github.io/maskprop/.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, background): “Interactive image segmentation approaches have been widely applied to annotate large-scale image datasets, which drive the success of deep models in various applications, including video understanding [5,49], self-driving [7], and medical imaging [32, 41].”
Is Space-Time Attention All You Need for Video Understanding? (2021, background): “However, in the video domain, 2D or 3D convolutions still represent the core operators for spatiotemporal feature learning across different video tasks (Feichtenhofer et al., 2019a; Teed & Deng, 2020; Bertasius & Torresani, 2020).”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “And like image segmentation, most existing methods are specialist models comprising multiple stages with pipe-lined frameworks, e.g., track-detect [67, 57, 36, 6, 44], clip-match [3, 5], propose-reduce [37].”
Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Most VIS approaches [52], [93], [94], [95] focus on learning instance-wised spatial, temporal relation, and featurefusion.”
Associating Objects with Transformers for Video Object Segmentation (2021, background): “As to related areas of VOS, the simple yet effective identification mechanism should also be promising for many tasks requiring multi-object matching, such as interactive video object segmentation [14, 17], video instance segmentation [1, 26, 31], and multi-object tracking [15, 27, 30].”
Video Instance Segmentation using Inter-Frame Communication Transformers (2021, methodology): “Our spatial decoder is of instance-agnostic design which gets highly efﬁcient than decoders of instance-speciﬁc designs [13, 11, 12, 10] as the number of detected instances increases.”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, background): “MaskProp [4] is also a video extension of Mask R-CNN which first propagates the predicted masks of each frame to adjacent frames and then matches the clip-level masks for tracking.”
Anticipative Video Transformer (2021, background): “Nonetheless, in addition to AVT-b, AVT is compatible with other video backbones, including those based on 2D CNNs [77, 88], 3D CNNs [12, 20, 84], or fixed feature representations based on detected objects [5, 6] or visual attributes [61].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “MaskProp [226] CVPR Instance Mask R-CNN Instance Feature Propagation YouTube-VIS [82]”