Learning Video Object Segmentation from Static Images

By Federico Perazzi, Anna Khoreva, Rodrigo Benenson, Bernt Schiele, Alexander Sorkine-Hornung. 2016

622 citations recorded by the source corpus.

Inspired by recent advances of deep learning in instance segmentation and object tracking, we introduce the concept of convnet-based guidance applied to video object segmentation. Our model proceeds on a per-frame basis, guided by the output of the previous frame towards the object of interest in the next frame. We demonstrate that highly accurate object segmentation in videos can be enabled by using a convolutional neural network (convnet) trained with static images only. The key component of our approach is a combination of offline and online learning strategies, where the former produces a refined mask from the previous frame estimate and the latter allows to capture the appearance of the specific object instance. Our method can handle different types of input annotations such as bounding boxes and segments while leveraging an arbitrary amount of annotated frames. Therefore our system is suitable for diverse applications with different requirements in terms of accuracy and efficiency. In our extensive evaluation, we obtain competitive results on three different datasets, independently from the type of input annotation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “The ORBIT dataset can be used to explore a wide set of real-world recognition tasks from continual learning [27, 28] to video segmentation [25, 34, 29].”
Occluded Video Instance Segmentation: A Benchmark (2021, background): “According to whether to provide the mask for the first frame, VOS can be divided into semi-supervised and unsupervised scenarios.”
Iteratively Trained Interactive Segmentation (2018, background): “Many recent works [4, 15, 23, 26] focus on segmenting objects in videos since such object annotations are expensive.”
One-Shot Video Object Segmentation (2016, background): “A coeval work [21], also using deep learning for video object segmentation, learns to refine the mask frame by frame.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “The task of video object segmentation requires segmenting foreground objects in a class-agnostic fashion [43, 21, 36, 38], often by leveraging ground truth masks available for the first frame during inference [6, 32, 10, 19, 37].”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, background): “In the semi-supervised scenario, where the ground-truth mask is available at least in a single frame, existing methods [2, 24, 35, 32, 29, 22] propagate the ground-truth object mask to the rest of frames in a video.”
Online Model Distillation for Efficient Video Inference (2018, background): “Video object segmentation: Solutions to video object segmentation (VOS) leverage online adaptation of highcapacity deep models to a provided reference segmentation in order to propagate instance masks to future frames [32, 47, 44, 4].”
Online Adaptation of Convolutional Neural Networks for Video Object Segmentation (2017, methodology): “Current state of the art methods use post-processing steps such as boundary snapping [7], or conditional random ﬁeld (CRF) smoothing [24, 35] to improve the contours.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “Level Core Architecture Flow Technical Feature Training Dataset 2016 Clockwork [205] ECCVSemantic FCN ✓ Faster Segmentation Cityscapes [206]/YouTube-Objects [73] FSO [207] CVPRSemantic FCN + Dense CRF ✓ Temporal Feature Aggregation Cityscapes [206]/CamVid [208] JFS [209] ECCVSemantic FCN ✓ Temporal ”
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks (2019, methodology): “Implementation Details: Following [44, 55], both static data from image salient object segmentation datasets, MSRA10K [8], DUT [72], and video data from the training set of DAVIS16 are iteratively used to train our model.”
Instance Embedding Transfer to Unsupervised Video Object Segmentation (2018, methodology): “MSK [21] trains a propagation network with synthetic transformations of still images and applies the same technique for online fine-tuning.”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “In contrast, tracking-based approaches [39,52,10,22,5,35,63,19,47,20,56] perform frame-to-frame propagation and are thus efficient at test-time.”
SwiftNet: Real-time Video Object Segmentation (2021, background): “Nonetheless, although pursued in fruitful endeavors [2, 22, 35, 6, 17, 29, 11], real-time VOS remains unsolved, as object variation over-time poses heavy demands for sophisticated object modeling and matching computations.”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, background): “The availability of such dataset introduced the ﬁrst wave of deep learning based methods in the ﬁeld [4], [5], [6], [7], [8], [9], [10], [11], [12], [13] that considerably improved the state of the art.”
Learning About Objects by Learning to Interact with Them (2020, methodology): “forms of supervision such as CNNs pre-trained for other tasks [35, 58, 57], supervised object proposal models [19, 11, 34], ﬁxation data as annotation [61], or instance segmentation for single frames [46]. Some of these approaches are considered unsupervised in that there is no supervision across th”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, methodology): “ learning. ‘+’ means we use ResNet-101 as feature extractor instead of ResNet-50 Methods J M F M G M MaskRNN [14] 45.5 - OSMN [46] 52.5 57.1 54.8 FAVOS [9] 54.6 61.8 58.2 VideoMatch [15] 56.5 - - MSK [28] 63.3 67.2 65.3 RGMP [43] 64.8 68.6 66.7 FEELVOS [38] 65.9 72.3 69.1 DyeNet [21] 67.3 71.0 69.1 ”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, methodology): “Another approach is to learn to propagate the segmentation mask from one frame to the next us-ing optical ﬂow as done by MaskTrack [28].”