Fast Video Object Segmentation by Reference-Guided Mask Propagation

By Seoung Wug Oh, Joon-Young Lee, Kalyan Sunkavalli, Seon Joo Kim. 2018

414 citations recorded by the source corpus.

We present an efficient method for the semi-supervised video object segmentation. Our method achieves accuracy competitive with state-of-the-art methods while running in a fraction of time compared to others. To this end, we propose a deep Siamese encoder-decoder network that is designed to take advantage of mask propagation and object detection while avoiding the weaknesses of both approaches. Our network, learned through a two-stage training process that exploits both synthetic and real data, works robustly without any online learning or post-processing. We validate our method on four benchmark sets that cover single and multiple object segmentation. On all the benchmark sets, our method shows comparable accuracy while having the order of magnitude faster runtime. We also provide extensive ablation and add-on studies to analyze and evaluate our framework.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “According to whether to provide the mask for the first frame, VOS can be divided into semi-supervised and unsupervised scenarios.”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, background): “A variety of methods have tackled the unsupervised ( i.e ., no object annotation available during testing) [73, 71, 27, 44, 95, 96, 48] and the semi-supervised ( i.e ., ﬁrst frame annotation available during testing) [12, 54, 49, 11, 53, 98, 45, 18, 63, 105, 17, 15, 102, 100] setting.”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, background): “In contrast, tracking-based approaches [39,52,10,22,5,35,63,19,47,20,56] perform frame-to-frame propagation and are thus efficient at test-time.”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “r (Eq.(5)) and write controller (Eq.(8)) are all implemented using ConvGRU[2], with 11 convolutional kernels. The project function f P (Eq.(1)) is also realized with 11 convolutional layer. Similar to[59], the readout function f R (Eq.(9)) is implemented with a decoder network, which consists of fou”
Learning Video Object Segmentation From Unlabeled Videos (2020, methodology): “Propagation based methods [18, 67, 71] rely on the segments of the previous frames and work in a frame-by-frame manner.”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Time per frame (seconds) J & F Ours OSMN [75] Ours (fastest) Ours (Fine-tuned) SiamMask [66] PReMVOS [44] STM-VOS [52] DyeNet [39] FEELVOS [63] OSVOS-S [47] RGMP [71] CINM [3] VideoMatch [29] OnAVOS [64] FAVOS [12] OSVOS [8]”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, neutral): “OSVOS-S [42], RGMP [44], RVOS [54], FEELVOS [56], OnAVOS [57], Video Colorization [59], SiamMask [61], CycleTime [64], RANet [65], OSMN [73], with bounding boxes throughout the video; the other aims for more ﬁne-grained tracking, i.e . relocalize the objects with pixel-level segmentation masks,…”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “Oﬄine methods alleviate this issue and reduce runtime [35, 32, 29, 22].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “Conditioning propagation on the initial frame-mask pair [127], [141], [148] seems a feasible solution to this.”
VideoMatch: Matching based Video Object Segmentation (2018, background): “tracking [9], pixel-wise metric learning [7], or network modulation [56, 38].”
Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Faster approaches have been proposed such as a more efficient online learning algorithm [8, 6, 7], MRF graph inference [51], temporal CNN [52], capsule routing [53], tracking [11, 13, 15, 54, 55, 56, 57], embedding learning [10, 58, 59] and space-time matching [17, 18, 19, 20].”
Associating Objects with Transformers for Video Object Segmentation (2021, methodology): “‡ : timing extrapolated from single-object speed assuming linear scaling in the number of objects. randomly applying multiple image augmentations [61].”
Fast Online Object Tracking and Segmentation: A Unifying Approach (2018, background): “Recently, there has been an increasing interest in the VOS community towards faster methods [40, 63, 10, 9, 24, 23].”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, methodology): “7 RGMP [1] X 51.”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, methodology): “…methods that pushed the boundaries of video object segmentation not only in terms of accuracy [14], [15], [16], [17], [18], [19], [20] but also in terms of computational efﬁciency, giving rise to fast methods that can even operate online [21], [22], [23], [24], [25], [26], [27], [28], [29].”
Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey (2020, methodology): “In another study [71], the authors have used ResNet [10] based Siamese Encoder with Global Convolutional Block for video object segmentation.”
Video Panoptic Segmentation (2020, methodology): “It combines several existing tasks: video object segmentation [1,3,4,27,30,35,36,39] and video object detection [9,42,43], and aims at simultaneous detection, segmentation, and tracking of instances in videos.”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “to common approaches for Video Object Segmentation, which only process one instance at a time [63,95].”