A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation

By Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus H. Gross, Alexander Sorkine-Hornung. 2016

2,383 citations recorded by the source corpus.

Over the years, datasets and benchmarks have proven their fundamental importance in computer vision research, enabling targeted progress and objective comparisons in many fields. At the same time, legacy datasets may impend the evolution of a field due to saturated algorithm performance and the lack of contemporary, high quality data. In this work we present a new benchmark dataset and evaluation methodology for the area of video object segmentation. The dataset, named DAVIS (Densely Annotated VIdeo Segmentation), consists of fifty high quality, Full HD video sequences, spanning multiple occurrences of common video object segmentation challenges such as occlusions, motionblur and appearance changes. Each video is accompanied by densely annotated, pixel-accurate and per-frame ground truth segmentation. In addition, we provide a comprehensive analysis of several state-of-the-art segmentation approaches using three complementary metrics that measure the spatial extent of the segmentation, the accuracy of the silhouette contours and the temporal coherence. The results uncover strengths and weaknesses of current approaches, opening up promising directions for future works.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Features by Watching Objects Move (2016, methodology): “Figure 7 shows the output of the ConvNet on frames from the DAVIS [33], FBMS [28] and VSB [11] datasets, which were not used in training.”
SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, methodology): “We report results on ﬁve benchmarks: GrabCut [40], Berkeley [37], SBD [20], DAVIS [39], and Pascal VOC [14].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, methodology): “We extend previous VOS efforts [31, 41, 43] by tackling 2.”
FocalClick: Towards Practical Interactive Image Segmentation (2022, methodology): “GrabCut [34] Berkeley [31] SBD [15] DAVIS [33] Method Train Data NoC 85 NoC 90 NoC 90 NoC 85 NoC 90 NoC 85 NoC 90”
Reviving Iterative Training with Mask Guidance for Interactive Segmentation (2021, methodology): “For the DAVIS [19] dataset, we follow the evaluation protocol described in [1] and select 345 randomly sampled frames of video sequences.”
Occluded Video Instance Segmentation: A Benchmark (2021, methodology): “As the first video object segmentation dataset, DAVIS [62,7] contains 150 videos and 376 densely annotated objects.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “Since the dawn of machine learning for computer vision, datasets have been curated to train models, for tasks from classification [2,3] to detection [4,5], captioning [6,7] and segmentation [8,9].”
Iteratively Trained Interactive Segmentation (2018, methodology): “DAVIS [22] is a dataset for video object segmentation.”
Deep Extreme Cut: From Extreme Points to Object Segmentation (2017, methodology): “Figure 6 (left) shows the performance of OSVOS in DAVIS-2016 [27] trained on the ground truth mask ( ) or the masks generated by DEXTR from extreme points ( ).”
A General Framework for Uncertainty Estimation in Deep Learning (2019, methodology): “We perform experiments on the Davis 2016 dataset [35], which has high quality videos with moving objects, as well as pixel-level object masks annotations.”
Tracking Emerges by Colorizing Videos (2018, background): “The attributes come from Table 1 of [60].”
YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark (2018, methodology): “Even though simple, such an online learning or one-shot learning scheme achieves top performance on video object segmentation benchmarks [15,16].”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, methodology): “Many recent datasets extend the task from image to video: DAVIS [29], YouTube-VOS (YTVOS) [47], MOTS [40], YouTube-VIS (YTVIS) [48], TAO [7].”
One-Shot Video Object Segmentation (2016, methodology): “Each column shows: the original method without boundary snapping (-BS), without pre-training the parent network on DAVIS (-PN), or without performing the oneshot learning on the specific sequence (-OS).”
Learning Video Object Segmentation from Static Images (2016, methodology): “DAVIS [34] consists of 50 high quality videos, totaling 3 455 frames.”
Video Instance Segmentation (2019, methodology): “For example, video object segmentation [3, 22, 23] aims at segmenting and tracking objects in videos, but does not require recognition of object categories.”
Video Salient Object Detection via Fully Convolutional Networks (2017, methodology): “We also report our performance on the newly developed DAVIS dataset, which is one of the most challenging video segmentation benchmarks.”
A Generalist Framework for Panoptic Segmentation of Images and Videos (2022, methodology): “Similar results are obtained for Cityscape, the details of which are given in Appendix C. Table 2 compares Pix2Seq-D to state-of-the-art methods on unsupervised video object segmentation on DAVIS, us-ing the standard J & F metrics [46].”