Efficient Video Object Segmentation via Network Modulation

By Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, Aggelos K. Katsaggelos. 2018

359 citations recorded by the source corpus.

Video object segmentation targets segmenting a specific object throughout a video sequence when given only an annotated first frame. Recent deep learning based approaches find it effective to fine-tune a general-purpose segmentation model on the annotated frame using hundreds of iterations of gradient descent. Despite the high accuracy that these methods achieve, the fine-tuning process is inefficient and fails to meet the requirements of real world applications. We propose a novel approach that uses a single forward pass to adapt the segmentation model to the appearance of a specific object. Specifically, a second meta neural network named modulator is trained to manipulate the intermediate layers of the segmentation network given limited visual and spatial information of the target object. The experiments show that our approach is 70× faster than fine-tuning approaches and achieves similar accuracy. Our model and code have been released at https://github.com/linjieyangsc/video_seg.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Hsuan Yang
3. Dacheng Tao
4. Xiaogang Wang
5. Yi Yang
6. Alan L. Yuille
7. Philip H. S. Torr
8. Chunhua Shen

Citations in the corpus

• Tracking Emerges by Colorizing Videos (2018, methodology): “Supervised Models: To analyze the gap between our self-supervised model and fully supervised approaches, we also consider the best available supervised approaches [46,47].”
• Semantics Disentangling for Text-To-Image Generation (2019, methodology): “These parameters are used to control the behavior of the main network for the tasks such as image stylization [15], visual reasoning [27], video segmentation [38], question answering [5] and etc.”
• YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “Most of these methods [7,8,10,11,9] build their models based on an image segmentation network and do not involve sequential modeling.”
• Slimmable Neural Networks (2018, methodology): “Conditional normalization is widely used in tasks including style transfer (Dumoulin et al., 2016; Li et al., 2017a; Huang & Belongie, 2017; Li et al., 2017b), image recognition (Li et al., 2016b; Yang et al., 2018) and many others (Perez et al., 2017b;a).”
• Video Instance Segmentation (2019, methodology): “We evaluate two recent video object segmentation algorithms OSMN [35] and FEELVOS [31].”
• A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “01 AFB-URR [154] 74.6 73.0 76.1 3.8 FAVOS [126] 58.2 54.6 61.8 0.56 Fasttan [155] 75.9 72.3 79.4 y7 MAST [160] 65.5 63.3 67.6 5.1 STM-Cycle [166] 71.7 68.7 74.7 38 CRW [174] 67.6 64.5 70.6 7.3 QMA [167] 71.9 - - 6.3 RGMP [127] 66.7 64.8 68.6 7.7 Fasttmu [156] 70.6 69.1 72.1 9.7 OSMN [128] 54.8 52.5 ”
• Learning Correspondence From the Cycle-Consistency of Time (2019, neutral): “0 Fully Supervised [81, 7] X 55.”
• VideoMatch: Matching based Video Object Segmentation (2018, background): “tracking [9], pixel-wise metric learning [7], or network modulation [56, 38].”
• Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation (2021, methodology): “Faster approaches have been proposed such as a more efficient online learning algorithm [8, 6, 7], MRF graph inference [51], temporal CNN [52], capsule routing [53], tracking [11, 13, 15, 54, 55, 56, 57], embedding learning [10, 58, 59] and space-time matching [17, 18, 19, 20].”
• SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation (2021, background): “T w o S ta g e OSMN MaskProp [44] ResNet-50 - 23.”
• SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation (2020, background): “…and recall. method category AP AP@0.5 AP@0.75 AR@1 AR@10 OSMN [49] mask propagation 23.4 36.5 25.7 28.9 31.1 FEELVOS [41] mask propagation 26.9 42.0 29.7 29.9 33.4 OSMN [49] track-by-detect 27. the state-of-the-art MaskTrack R-CNN [48] with an absolute gain of 2.2% in terms of mask accuracy (AP).”
• FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, background): “OSMN [40], FAVOS [7], PML [6], and VideoMatch [17] all achieve very high speed and effectively bypass finetuning, but we show that the proposed FEELVOS produces significantly better results.”
• A Generative Appearance Model for End-To-End Video Object Segmentation (2018, background): “Recent works have posed video object segmentation as a feedforward mask-refinement process [23,31,34], where the previous mask prediction is adapted to fit the target in the current frame using a convolutional neural network.”
• Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation (2021, neutral): “From Table 6, we can see that among existing fast video object segmentation (VOS) methods, OSMN [44] with ‘track-by-detect’ and FEELVOS [37] achieve relatively lower mask AP scores.”