Scaling Egocentric Vision: The EPIC-KITCHENS Dataset

By Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price and 1 more. 2018

1,366 citations recorded by the source corpus.

First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict nonscripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labeled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens. Dataset and Project page: this http URL

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

RareAct: A video dataset of unusual interactions (2020, background): “The EPIC-KITCHENS video dataset [2] is the closest video dataset related to ours, where actions are also annotated as a combination of a verb and a noun.”
A Short Note on the Kinetics-700 Human Action Dataset (2019, background): “The additional classes for Kinetics-700 over Kinetics-600 were partly sourced from the lists of actions (or verbs) in recent human action datasets, such as EPIC-Kitchens [4] and AVA [5].”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “KITCHENS [7]), are on the scale of tens to hundreds of thousands of such pairs that have been annotated manually.”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Lack of sufficient data however, consistently remains a bottleneck – motivating researchers to construct new larger-scale egocentric video datasets including EPIC- Kitchens (Damen et al. 2018), Charades-Ego (Sigurdsson et al. 2018), and Ego4D (Grauman et al. 2021).”
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, background): “Something-Something [12] and EPIC-Kitchens [7] are video datasets collected by users with mobile and head-mounted cameras, respectively, but are focused on action recognition based on many examples and “action captions”.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “Learning from instructional videos is challenging due to its data complexity across various tasks [6, 1, 51, 26].”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “2, 3, 17 [12] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “Similarly, we uniformly sample one frame per three seconds in EPIC-KITCHEN [8] videos across 37 participants, then filter out frames that contain hand-object, and finally manually select the interesting frames to label.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, methodology): “Inspired by the pause-andtalk narrator [44], annotators are asked to watch a 5 minute clip of video, summarize it with a few sentences, and then re-watch, pausing repeatedly to write a sentence about each thing the camera wearer does.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, background): “, [8, 30, 31], uses strong supervision in the form of direct labels, including a lot of work that focuses on similar objectives [9, 11, 14].”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, neutral): “Dataset Duration Samples Segments Type of Task Video Source Hierarchical Classes Year MPII [35] 9h,48m 44 5,609 cooking activities self-collected 7 - 2012 YouCook [14] 2h,20m 88 cooking activities YouTube 7 - 2013 50Salads [40] 5h,20m 50 966 cooking activities self-collected 7 - 2013 Breakfast [28] ”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, methodology): “Interestingly, on EPIC-Kitchens, pre-training either using IG-Kinetics-65M or Kinetics (referred to as baseline) yield similar gains with the increase in model capacity.”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, background): “Many real-world applications such as object search [27, 30], instance registration [50], human-object interaction modeling [14] and human activity understanding [6] require such open-world prediction abilities, e.”
Understanding Human Hands in Contact at Internet Scale (2020, background): “Of the many works in video human-object interaction [11, 39, 33, 24, 40], ours is most clearly related to VLOG [14], which also gathers data of interaction, and AVA [18], which investigates atomic (i.e., base-level) actions.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “AVA [26] 108 504K 80 - X 49 Charades [66] 82 10K 157 X 37 EPIC-Kitchen [15] 55 125 X 331 DALY [75] 31 8K 10 X X 41 3.”
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings (2019, methodology): “Our endeavour has been facilitated by the recent release of open vocabulary narrations on the EPIC dataset [6] which we note is the only ﬁne-grained dataset to do so.”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Speciﬁcally, we consider the Deep Multimodal Regressor (DMR) proposed in [9], the Anticipation Temporal Segment Network (ATSN) of [10], the anticipation Temporal Segment Network trained with verb-noun Marginal Cross Entropy Loss (MCE) described in [11], and the Encoder-Decoder LSTM (ED) introduced…”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “More general interaction information can be learned from large human datasets [18,19,21,40,62,67], to predict object information [30, 135] (RGB & 3D) [10], graphs [24] or environment information [28, 80] such as heatmaps [39, 79].”