One-Shot Video Object Segmentation

By Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, L. Leal-Taix'e, Daniel Cremers, Luc Van Gool. 2016

984 citations recorded by the source corpus.

This paper tackles the task of semi-supervised video object segmentation, i.e., the separation of an object from the background in a video, given the mask of the first frame. We present One-Shot Video Object Segmentation (OSVOS), based on a fully-convolutional neural network architecture that is able to successively transfer generic semantic information, learned on ImageNet, to the task of foreground segmentation, and finally to learning the appearance of a single annotated object of the test sequence (hence one-shot). Although all frames are processed independently, the results are temporally coherent and stable. We perform experiments on two annotated video segmentation databases, which show that OSVOS is fast and improves the state of the art by a significant margin (79.8% vs 68.0%).

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, background): “Orcun Cetintas | AutoML One-Shot Video Object Segmentation [1] First frame Ground truth”
EDTER: Edge Detection with Transformer (2022, background): “Edge detection is one of the most fundamental problems in computer vision and has a wide variety of applications, such as image segmentation [8, 23, 39, 44, 45, 47], object detection [23], and video object segmentation [5, 57, 59].”
Iteratively Trained Interactive Segmentation (2018, methodology): “Following [2], we refine the results obtained by OSVOS [4] and report the segmentation quality at 1, 4 and 10 clicks in Table 2.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “Class-balancing the loss gives more importance to the less frequent classes, and has been successful in various tasks [39, 5].”
Tracking Emerges by Colorizing Videos (2018, methodology): “Supervised Models: To analyze the gap between our self-supervised model and fully supervised approaches, we also consider the best available supervised approaches [46,47].”
Generalizing from a Few Examples (2019, methodology): “FSL has also been successfully used in video applications, including motion prediction [50], video classification [164], action localization [152], person re-identification [148], event detection [151], and object segmentation [21].”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “Most of these methods [7,8,10,11,9] build their models based on an image segmentation network and do not involve sequential modeling.”
Learning Video Object Segmentation from Static Images (2016, methodology): “osed for instance segmentation [17,32,33,49]. Our approach outputs per-frame instance segmentation using a convnet architecture, inspired by works from other domains like [6,40,49]. A concurrent work [5] also exploits convnets for video object segmentation. Differently from our approach their segmen”
Video Instance Segmentation (2019, background): “For example, video object segmentation [3, 22, 23] aims at segmenting and tracking objects in videos, but does not require recognition of object categories.”
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation (2019, background): “The task of video object segmentation requires segmenting foreground objects in a class-agnostic fashion [43, 21, 36, 38], often by leveraging ground truth masks available for the ﬁrst frame during inference [6, 32, 10, 19, 37].”
Learning About Objects by Learning to Interact with Them (2020, background): “[43, 40, 8] require annotations for the first or few frames of the video.”
Online Model Distillation for Efficient Video Inference (2018, methodology): “Video object segmentation: Solutions to video object segmentation (VOS) leverage online adaptation of highcapacity deep models to a provided reference segmentation in order to propagate instance masks to future frames [35, 51, 48, 5].”
The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation (2019, background): “The availability of such dataset introduced the ﬁrst wave of deep learning based methods in the ﬁeld [4], [5], [6], [7], [8], [9], [10], [11], [12], [13] that considerably improved the state of the art.”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, methodology): “FEELVOS achieves a J score of 82.1% which is even better than the results of the finetuning based methods OSVOS [1] and OnAVOS [35].”
SegFlow: Joint Learning for Video Object Segmentation and Optical Flow (2017, methodology): “Compared to other methods such as OFL (30 seconds per frame for optical flow generation and 90 seconds per frame for optimization), MSK (12 seconds per frame) and OSVOS (more than 10 seconds per frame at its best performance), our method is faster and can output an additional result of optical flow.”
Latent Backdoor Attacks on Deep Neural Networks (2019, background): “In a deep learning context, transfer learning has been shown to be effective in vision [5, 11, 42, 43], speech [13, 20, 24, 51], and text [22, 34].”
The 2017 DAVIS Challenge on Video Object Segmentation (2017, methodology): “We also add OSVOS [5] as baseline: each object of the sequence segmented independently and merged together naively.”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “The idea of fine-tuning was first introduced in [18], where only the initial image-mask pair is used for training an online,”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “…F Mean Time OSMN [40] 74.0 72.9 0.14 s SFL [5] 74.8 74.5 7.9 s PML [3] 75.5 79.3 0.27 s S2S (+YV) [38] 79.1 - 9 s MSK [27] 79.7 75.4 12 s OSVOS [2] 79.8 80.6 9 s LCT [16] 80.5 77.6 - MaskRNN [12] 80.7 80.9 - VidMatch [13] 81.0 - 0.32 s FEEL (+YV) [34] 81.1 82.2 0.45 s RGMP [25] 81.5 82.0 0.13…”
Data Augmentation Using Learned Transformations for One-Shot Medical Image Segmentation (2019, background): “labeled datasets of supplementary information such as object appearances [11], or incorporate additional information such as human input [60].”