Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

By Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price and 1 more. 2020

719 citations recorded by the source corpus.

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Multiview Transformers for Video Recognition (2022, methodology): “Following the standard protocol [15], we report Top-1 action-, verb- and noun-accuracies with action accuracy being the primary metric.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “Like with EPIC-Kitchens, we also evaluate how much audio can help by masking the audio inputs.”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “Several egocentric video datasets have been created over the years, including [28, 29, 30].”
Masked Visual Pre-training for Motor Control (2022, methodology): “Specifically, we use images from the egocentric Epic Kitchens dataset (Damen et al., 2018; 2021), the YouTube 100 Days of Hands dataset (Shan et al., 2020), and the crowd-sourced Something-Something dataset (Goyal et al., 2017b).”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, methodology): “Epic-Kitchens-100 (EK100) : The EK100 [14] dataset consists of 100 hours of video in egocentric perspective recording a person interacting with a variety of objects in the kitchen.”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “EPIC-KITCHENS [446], [450] are large benchmark datasets suitable for fusion and co-learning of different modalities.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Approaches also track human poses, especially hands [14,18,61,62,96,101,120].”
SAM 2: Segment Anything in Images and Videos (2024, neutral): “…(Haucke et al., 2022), STREETS (Snyder & Do, 2019), TimberSeg (Fortin et al., 2022), TrashCan (Hong et al., 2020), VISOR (Darkhalil et al., 2022; Damen et al., 2022), WoodScape (Yogamani et al., 2019), PIDRay (Wang et al., 2021a), ZeroWaste-f (Bashkirova et al., 2022), and IBD (Chen et al.,…”
Data augmentation techniques for the Video Question Answering task (2020, background): “ until very recently,when Fan released the EgovQA dataset in [6]. Yet, considering the recent advancements in several elds of the egocentric vision, such as action recognition and action anticipation [3,4], Egocentric VideoQA also plays a primary role in the understanding of the complex interactions”
Egocentric Video-Language Pretraining (2022, methodology): “5 M HowTo100M [10] instructional 134 K 136 M 136 M 3rd-person view Charades-Ego [17] home 34 30 K 30 K UT-Ego [18] diverse 37 11 K 11 K Disneyworld [19] disneyland 42 15 K 15 K EPIC-KITCHENS-100 [20] kitchen 100 90 K 90 K EgoClip diverse 2 .”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, positive): “Using the Epic-Kitchens-100 (EK100) benchmark (Damen et al., 2022), we demonstrate that V-JEPA 2 action anticipation performance increases consistently with model size.”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, background): “Existing real-world benchmarks rely on one of the following data sources: (i) Videos collected from the web or repositories like Youtube, e.g. Kinetics [36], ActivityNet [9], VGGSound [11], HVU [18], ActivityNet-QA [56], tGIFQA [33]; (ii) Videos collected on demand, filmed by volunteers doing arbitr”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “In particular, we train a 307M parameter vi42 sion encoder [8] on a massive collection of images coming from ImageNet [10], Epic Kitchens [16], 43 Something Something [11], 100 Days of Hands [12], and Ego4D [14].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “As shown in Table 8, we observe that a model trained with larger data (EK100) outperforms a model trained on EK55 on both tasks.”