The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines

By Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price and 1 more. 2020

366 citations recorded by the source corpus.

Since its introduction in 2018, EPIC-KITCHENS has attracted attention as the largest egocentric video benchmark, offering a unique viewpoint on people’s interaction with objects, their attention, and even intention. In this paper, we detail how this large-scale dataset was captured by 32 participants in their native kitchen environments, and densely annotated with actions and object interactions. Our videos depict nonscripted daily activities, as recording is started every time a participant entered their kitchen. Recording took place in four countries by participants belonging to ten different nationalities, resulting in highly diverse kitchen habits and cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.2K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. We introduce new baselines that highlight the multimodal nature of the dataset and the importance of explicit temporal modelling to discriminate fine-grained actions (e.g., ‘closing a tap’ from ‘opening’ it up).

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[42] Dima Damen, Hazel Doughty, Giovanni Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.”
Learning Open-World Object Proposals without Learning to Classify (2021, methodology): “In addition, we study cross-dataset generalization from COCO to RoboNet [10] and EpicKitchens [11].”
Connecting Vision and Language with Localized Narratives (2019, background): “In the video domain, EPIC-KITCHENS [15] contains videos of daily kitchen activities collected with a head-mounted camera.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “Task-oriented, instructional language focuses on objects and interactions between objects and actors, as seen in instructional datasets (Damen et al., 2020; Koupaee and Wang, 2018), as a function of the inextricable relationship between language and objects (Quine, 1960).”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, methodology): “While some previous datasets such as EPICKITCHENS [13, 14] and ADL [46] have been labeled with bounding box annotations, these datasets have not been explicitly labeled for the EHOI detection task indicating relationships between labeled objects and actions, hence preventing the development of EHOI ”
Data augmentation techniques for the Video Question Answering task (2020, background): “Yet, considering the recent advancements in several fields of the egocentric vision, such as action recognition and action anticipation [3,4], Egocentric VideoQA also plays a primary role in the understanding of the complex interactions of the first-person videos.”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “EPIC-Kitchens and its extension [13, 14, 15] are a series of egocentric datasets focused on unscripted ac-tivites related to kitchens.”
Human Hands as Probes for Interactive Object Understanding (2021, methodology): “We train our models on in-the-wild videos from EPICKITCHENS [10].”
PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, background): “Similar to real-world egocentric video [15], our synthetic ego-centric views yield particularly challenging motion trajectories.”
3rd Continual Learning Workshop Challenge on Egocentric Category and Instance Level Object Understanding (2022, background): “The most relevant to our dataset are those containing daily life objects, such as Objectron [55], CO3D [56], Epic-Kitchens [57], and Ego4D [58].”
OtterHD: A High-Resolution Multi-modality Model (2023, methodology): “Specifically, the collection comprises 172 QA pairs from 108 images in EpicKitchen [14], 80 QAs from 38 images in Ego4D [22], and 31 QA pairs from 20 images in VidOR [46].”