YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark

By Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, Thomas S. Huang. 2018

678 citations recorded by the source corpus.

Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatialtemporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 4,453 YouTube video clips and 94 object categories. This is by far the largest video object segmentation dataset to our knowledge and has been released at this http URL We further evaluate several existing state-of-the-art video object segmentation algorithms on this dataset which aims to establish baselines for the development of new algorithms in the future.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, methodology): “[82] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang.”
SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, background): “It has been widely applied to annotate large-scale image datasets, which drive the success of deep models in various applications, including video understanding [4, 47], self-driving [6], and medical imaging [29, 38].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, methodology): “We extend previous VOS efforts [31, 41, 43] by tackling 2.”
FocalClick: Towards Practical Interactive Image Segmentation (2022, methodology): “We also report the performance of our model trained on SBD [15], and a large combined dataset [2, 7, 13, 24, 26, 39, 45, 48].”
BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (2018, methodology): “BDD100K MOTS provides a MOTS dataset that is larger than the KITTI and MOTS Challenge datasets, with the number of annotations comparable with the large-scale YouTube VOS [34] dataset.”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, methodology): “Besides shifting the focus to the open-world setup, UVO has approximately 8x more videos compared with DAVIS [31], and 7x more mask (instance) annotations per video compared with YouTube-VOS [49] and YouTube-VIS [50].”
Video Instance Segmentation (2019, methodology): “YTO [13] FBMS [20] DAVIS [22, 23] YTVOS [34] YTVIS”
PseudoClick: Interactive Image Segmentation with Click Imitation (2022, background): “g, semantic object/instance segmentation [1,2], video understanding [3,4], autonomous driving [5,6,7], and medical image analysis [8,9].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “PASCAL VOC 2012 [103] + COCO [123] + ECSSD [118] + DAVIS17 [81]/YouTubeVOS [95] Fasttan [155] CVPR Faster RCNN Propagation-based COCO [123] + DAVIS17 [81]”
SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines (2019, background): “Instance-level Video Object Segmentation data has been made available even more recently [61, 79, 101] since it requires annotation of multiple objects per video.”
Suppress and Balance: A Simple Gated Network for Salient Object Detection (2020, background): “Youtube-VOS [60] is the latest large-scale dataset for the video object segmentation that consists of 4,453 videos annotated with multiple objects.”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, methodology): “In order to construct our dataset and to inform our model prior, we extract segmentation masks from YouTube-VOS [17, 18] and KITTI-MOTS [19–21].”
Online Model Distillation for Efficient Video Inference (2018, background): “Existing large-scale video datasets have been designed to support training high-quality models for various tasks, such as action detection [23, 41], object detection, tracking, and segmentation [33, 46], and consist of carefully curated, diverse sets of short video clips (seconds to a couple minutes”
Learning Joint Spatial-Temporal Transformations for Video Inpainting (2020, methodology): “ for ensuring spatial and temporal coherence. {CAP [18] synthesizes missing contents by a deep alignment network and a frame-based attention module. We ne-tune baselines multiple times on YouTube-VOS [32] and DAVIS [3] by their released models and codes and report their best results in this paper. W”
XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model (2022, methodology): “Table S3 tabulates our results on the YouTubeVOS [15] 2019 validation set.”
A Generative Appearance Model for End-To-End Video Object Segmentation (2018, background): “S2S [33] 64.4 71.0 55.5 OSVOS [2] 58 The impact of the multi-modal generative model is also analyzed qualitatively in ﬁg.”
Continuous Adaptation for Interactive Object Segmentation by Learning from Corrections (2019, background): “(2) YouTube -VOS [57], a large video object segmentation dataset.”
MAST: A Memory-Augmented Self-Supervised Tracker (2020, methodology): “We benchmark our model on two public benchmarks: DAVIS-2017 [48] and the current largest video segmentation dataset, YouTube-VOS [71].”