YouTube-VOS: Sequence-to-Sequence Video Object Segmentation

By Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian L. Price, Scott D. Cohen, Thomas S. Huang. 2018

550 citations recorded by the source corpus.

Learning long-term spatial-temporal features are critical for many video analysis tasks. However, existing video segmentation methods predominantly rely on static image segmentation techniques, and methods capturing temporal dependency for segmentation have to depend on pretrained optical flow models, leading to suboptimal solutions for the problem. End-to-end sequential learning to explore spatial-temporal features for video segmentation is largely limited by the scale of available video segmentation datasets, i.e., even the largest video segmentation dataset only contains 90 short video clips. To solve this problem, we build a new large-scale video object segmentation dataset called YouTube Video Object Segmentation dataset (YouTube-VOS). Our dataset contains 3,252 YouTube video clips and 78 categories including common objects and human activities (This is the statistics when we submit this paper, see updated statistics on our website). This is by far the largest video object segmentation dataset to our knowledge and we have released it at https://youtube-vos.org. Based on this dataset, we propose a novel sequence-to-sequence network to fully exploit long-term spatial-temporal information in videos for segmentation. We demonstrate that our method is able to achieve the best results on our YouTube-VOS test set and comparable results on DAVIS 2016 compared to the current state-of-the-art methods. Experiments show that the large scale dataset is indeed a key factor to the success of our model.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Occluded Video Instance Segmentation: A Benchmark (2021, background): “the large-scale video object segmentation dataset YouTubeVOS [63], the 2019 version of YouTube-VIS dataset contains a total of 2,883 videos, 4,883 instances, and 131k masks in 40 categories.”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “We believe it is because instance categories are used to split the seen and unseen subset in the current setting, comparing to [34] in which subsets are split using video-level categories.”
Shifting More Attention to Video Salient Object Detection (2019, background): “, video segmentation [74, 83], video captioning [57], video compression [27, 29], autonomous driving [91], robotic interaction [82], weakly supervised attention [95].”
Siam R-CNN: Visual Tracking by Re-Detection (2019, methodology): “Table 6 and Figure 10 show the results of our method on the YouTube-VOS 2018 benchmark [101] (474 videos, 26.”
Video Instance Segmentation 2019: A Winning Approach for Combined Detection, Segmentation, Classification and Tracking. (2019, methodology): “This VIS task was recently introduced in [28], with the release of the YouTube-VIS dataset (YT-VIS) which is the YouTube-VOS [27] dataset adapted to the VIS task for 40 different object categories.”
Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge (2021, background): “Building upon the large-scale video classification dataset YouTube-8M [1], YouTube-VOS contains 4,453 video clips and 7,755 objects.”
Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks (2019, methodology): “In [58], the video segmentation task was transformed into a”
URVOS: Unified Referring Video Object Segmentation Network with a Large-Scale Benchmark (2020, methodology): “We follow the original Youtube-12 VOS dataset [34] to split the data into training and validation sets.”
MOTS: Multi-Object Tracking and Segmentation (2019, methodology): “Furthermore, the YouTube-VOS dataset [59] is available and orders of magnitude larger than DAVIS.”
FEELVOS: Fast End-To-End Embedding Learning for Video Object Segmentation (2019, methodology): “FEELVOS (-YTB-VOS) denotes training without YouTube-VOS [39] training data.”
A Generative Appearance Model for End-To-End Video Object Segmentation (2018, methodology): “We perform extensive experiments on 3 datasets, including the recent large-scale YouTubeVOS dataset [32].”
Video Object Segmentation Using Space-Time Memory Networks (2019, methodology): “2 in Overall score on Youtube-VOS [38].”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, methodology): “S2S [11] proposed to model video object segmentation as a sequence-to-sequence problem and proceeds to exploit a temporal modeling module to enhance the temporal coherency of mask propagation.”
SwiftNet: Real-time Video Object Segmentation (2021, methodology): “Instead of utilizing merely the first and last frames for object modeling, incorporating all historical frames as reference help establish temporally-coherent object evolution [20, 37].”
Deformable Siamese Attention Networks for Visual Object Tracking (2020, methodology): “We use ResNet-50, pre-trained on ImageNet [9], as the backbone, and the whole networks are then fine-tuned on the training sets of COCO [27], YouTube-VOS [39], LaSOT [10] and TrackingNet [30].”
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation (2019, methodology): “Compared to the state-of-the-art method S2S, our model achieves competitive segmentation metrics with double speed.”
Video Instance Segmentation Tracking With a Modified VAE Architecture (2020, methodology): “Popular datasets for the VOS task include DAVIS [50], Seg-TrackV2 [38] and YouTube-VOS [64].”
Deep Video Inpainting (2019, methodology): “We use the foreground segmentation masks of the YouTube-VOS dataset [29].”