Online Adaptation of Convolutional Neural Networks for Video Object Segmentation

By Paul Voigtlaender, Bastian Leibe. 2017

417 citations recorded by the source corpus.

We tackle the task of semi-supervised video object segmentation, i.e. segmenting the pixels belonging to an object in the video using the ground truth pixel mask for the first frame. We build on the recently introduced one-shot video object segmentation (OSVOS) approach which uses a pretrained network and fine-tunes it on the first frame. While achieving impressive performance, at test time OSVOS uses the fine-tuned network in unchanged form and is not able to adapt to large changes in object appearance. To overcome this limitation, we propose Online Adaptive Video Object Segmentation (OnAVOS) which updates the network online using training examples selected based on the confidence of the network and the spatial configuration. Additionally, we add a pretraining step based on objectness, which is learned on PASCAL. Our experiments show that both extensions are highly effective and improve the state of the art on DAVIS to an intersection-over-union score of 85.7%.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “According to whether to provide the mask for the first frame, VOS can be divided into semi-supervised and unsupervised scenarios.”
Iteratively Trained Interactive Segmentation (2018, background): “Many recent works [4, 15, 23, 26] focus on segmenting objects in videos since such object annotations are expensive.”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “These algorithms are OSVOS [7], MaskTrack [8], OSMN [9], OnAVOS [35] and S2S [34].”
VideoMatch: Matching based Video Object Segmentation (2018, methodology): “We also compare to OSVOS [5], MSK [25], OnAVOS [53] and SFL [8], disabling their fine-tuning step.”
Learning Video Object Segmentation From Unlabeled Videos (2020, methodology): “As our method does not use any first-frame fine-tuning [6, 35] or online learning [55] technique, it is fast for inference.”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “Voigtlaen-der and Leibe [35] employ the idea of online adaptation and continuously update the detector using the intermediate outputs.”
SwiftNet: Real-time Video Object Segmentation (2021, background): “Additionally, time-consuming online fine-tuning are exploited in [2, 18, 31] to improve segmentation accuracy, which however is impractical for realtime application.”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, background): “In semi-supervised mode [9, 60, 46, 12, 21], the initial masks for objects of interest are provided in the ﬁrst frame.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “tion, many OTTA methods [490], [492], [511], [515], [516], [517] adopt pseudo labels generated at test time for model updates.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, background): “For example, when supervision for the first frame of a test video is available [43,55,13], or after annotating a subset of an image dataset [42,61].”
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation (2018, methodology): “MaskTrack and OnAVOS have better performance on unseen than seen categories.”
Online Model Distillation for Efficient Video Inference (2018, methodology): “Video object segmentation: Solutions to video object segmentation (VOS) leverage online adaptation of highcapacity deep models to a provided reference segmentation in order to propagate instance masks to future frames [32, 47, 44, 4].”
Fast Video Object Segmentation by Reference-Guided Mask Propagation (2018, methodology): “In the table, the runtimes of OSVOS [5] and OFL [38] are from [40] and [30], respectively.”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Many methods perform fine-tuning on the template masks [8, 47, 64, 39, 3, 44], which leads to strong results but is slow.”
AMP: Adaptive Masked Proxies for Few-Shot Segmentation (2019, methodology): “Our method is evaluated on the DAVIS-2016 benchmark [22], where our base network is a 2-stream Wide ResNet model similar to [36].”
Video Object Segmentation with Episodic Graph Memory Networks (2020, methodology): “OSMN SIMMASK FAVOS RVOS OSVOS AGAME OnAVOS RGMP Method [66] [51] [7] [48] [5] [17] [50] [59]”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, methodology): “OnAVOS [35, 34] and OSVOS-S [27] extend OSVOS by an online adaptation mechanism, and by semantic information from an instance segmentation network, respectively.”
Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting (2018, methodology): “, [32][2][12], the method is initialized manually by a segmentation mask in the first few frames, then the segmented object is tracked throughout the video sequence.”