Interactive Video Object Segmentation in the Wild

By Arnaud Benard, Michael Gygli. 2017

52 citations recorded by the source corpus.

In this paper we present our system for human-in-the-loop video object segmentation. The backbone of our system is a method for one-shot video object segmentation. While fast, this method requires an accurate pixel-level segmentation of one (or several) frames as input. As manually annotating such a segmentation is impractical, we propose a deep interactive image segmentation method, that can accurately segment objects with only a handful of clicks. On the GrabCut dataset, our method obtains 90% IOU with just 3.8 clicks on average, setting the new state of the art. Furthermore, as our method iteratively refines an initial segmentation, it can effectively correct frames where the video object segmentation fails, thus allowing users to quickly obtain high quality results even on challenging sequences. Finally, we investigate usage patterns and give insights in how many steps users take to annotate frames, what kind of corrections they provide, etc., thus giving important insights for further improving interactive video segmentation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Getting to 99% Accuracy in Interactive Segmentation (2020, methodology): “ck Embedding Click Fusion Feedback Fusion iFCN Xu et al. [6] Distance Map early none ITIS Mahadevan et al. [7] Gaussian early early DEXTR Maninis et al. [8] Gaussian early none VOS-wild Benard et al. [9] Gaussian early none FCTSFN Hu et al. [10] Distance Map late none IIS-LD Li et al. [11] Gaussian ”
Iteratively Trained Interactive Segmentation (2018, methodology): “[2] found that encoding clicks by Gaussians yields better results.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, methodology): “We use two standard metrics [56,35,10,38,33,11,29]: (1) IoU@k, the average intersection-over-union between the ground-truth and predicted segmentation masks, given k corrections per image, and (2) clicks@q%, the average number of corrections needed to reach an IoU of q% on every image (thresholded a”
Content-Aware Multi-Level Guidance for Interactive Instance Segmentation (2019, methodology): “The segmented results are useful for downstream applications such as image/video editing [6, 30], image-based medical diagnosis [50, 51], human-machine collaborative annotation [2], etc .”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “[142] introduce a novel Interactive Video Object Segmentation method based on [154].”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “To overcome such limitations, interactive approaches [1, 3, 4, 21] have recently been investigated to allow user interventions during inference.”