Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation

By Weiyao Wang, Matt Feiszli, Heng Wang, Du Tran. 2021

157 citations recorded by the source corpus.

Current state-of-the-art object detection and segmentation methods work well under the closed-world assumption. This closed-world setting assumes that the list of object categories is available during training and deployment. However, many real-world applications require detecting or segmenting novel objects, i.e., object categories never seen during training. In this paper, we present, UVO (Unidentified Video Objects), a new benchmark for openworld class-agnostic object segmentation in videos. Besides shifting the focus to the open-world setup, UVO is significantly larger, providing approximately 6 times more videos compared with DAVIS, and 7 times more mask (instance) annotations per video compared with YouTube-VO(I)S. UVO is also more challenging as it includes many videos with crowded scenes and complex background motions. We also demonstrated that UVO can be used for other applications, such as object tracking and super-voxel segmentation. We believe that UVO is a versatile testbed for researchers to develop novel approaches for open-world class-agnostic object segmentation, and inspires new research directions towards a more comprehensive video understanding beyond classification and detection.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, methodology): “We extend previous VOS efforts [31, 41, 43] by tackling 2.”
Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity (2022, methodology): “However, [54] provides a new benchmark for the problem while our work provides an approach.”
Learning Open-World Object Proposals without Learning to Classify (2021, background): “We want to ask, is it possible to combine the best of both worlds and “learn open-world (novel) object proposals?” This could potentially unlock learning-based proposals for promising applications including open-world detection [30] / segmentation [66], robot grasping [15], egocentric video understa”
Opening up Open World Tracking (2021, background): “Recent parallel work [80] focuses on labeling a variety of object classes in human-centric Kinetics400 dataset [34].”
Transformer-Based Visual Segmentation: A Survey (2023, unknown): “Meanwhile, open set segmentation [285] requires the model to output class agnostic masks and enhance the generality of segmentation models.”
SAM 2: Segment Anything in Images and Videos (2024, unknown): “Among these 17 zero-shot video datasets above, 9 of them (EndoVis, ESD, LVOSv2, LV-VIS, UVO, VOST, PUMaVOS, Virtual KITTI 2, and VIPSeg) have dense object segments annotated for every video frame.”
Segment Anything in High Quality (2023, methodology): “We compare HQ-SAM with SAM on a suite of 9 diverse segmentation datasets across different downstream tasks, where 7 out of them are under a zero-shot transfer protocol, including COCO [31], UVO [41], LVIS [14], HQ-YTVIS [20], BIG [6], COIFT [29] and HR-SOD [50].”