EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations
By Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard E. L. Higgins, Sanja Fidler, David F. Fouhey, Dima Damen. 2022
169 citations recorded by the source corpus.
We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting hands. VISOR introduces an annotation pipeline, AI-powered in parts, for scalability and quality. In total, we publicly release 272K manual semantic masks of 257 object classes, 9.9M interpolated dense masks, 67K hand-object relations, covering 36 hours of 179 untrimmed videos. Along with the annotations, we introduce three challenges in video object segmentation, interaction understanding and long-term reasoning. For data, code and leaderboards: http://epic-kitchens.github.io/VISOR
Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Jitendra Malik
- Luc Van Gool
- Trevor Darrell
- Cordelia Schmid
- Dacheng Tao
- Andrew Zisserman
- Andrea Vedaldi
- Xiaogang Wang
In Evaluation & Benchmarks
- Jitendra Malik
- Christoph Feichtenhofer
- Piotr Doll'ar
- Junting Pan
- Angjoo Kanazawa
- David F. Fouhey
- Georgios Pavlakos
- Ilija Radosavovic
Citations in the corpus
- Reconstructing Hands in 3D with Transformers (2023, methodology): “Hand pose datasets [52, 56] [9], Epic-Kitchens [12, 13], and Ego4D [18].”
- Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “More general interaction information can be learned from large human datasets [18,19,21,40,62,67], to predict object information [30, 135] (RGB & 3D) [10], graphs [24] or environment information [28, 80] such as heatmaps [39, 79].”
- SAM 2: Segment Anything in Images and Videos (2024, background): “VISOR (Darkhalil et al., 2022) contains egocentric videos in kitchen scenes with segments around hands and active objects.”