Ego4D: Around the World in 3,000 Hours of Egocentric Video

By Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu and 74 more. 2021

1,925 citations recorded by the source corpus.

We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of dailylife activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards, with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception. Project page: https://ego4d-data.org/

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “With the ever-increasing demand for on-device use cases over billions of client devices, such as teaching AI to understand the physical environment from the first-person perception [91] or personalizing AI tasks, the carbon footprint for on-device AI can add up to a dire amount quickly.”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “3.3-A. Socratic Egocentric Image Summaries.”
All in One: Exploring Unified Video-Language Pre-Training (2022, background): “Ego4d [14] is a egocentric dataset that has a large domain gap with our thirdview video from Youtube.”
EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, methodology): “In [15], the aim is to find the previous encounter of the object itself and [9] aims to ground the object in a single image.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “A major characteristic is that our dataset contains very diverse hand-object interaction activities and scenarios, where frames are sparsely sampled from nearly 1,000 videos in Ego4D [16], EPIC-KITCHEN [8], THU-READ [68], and from our own collected GoPro videos.”
Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “155 119 67,428 Ego4D [487] 2022 RGB,Au,Ac,etc.”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, background): “, 2022), audio (Piczak, 2015), and IMU (Grauman et al., 2022).”
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities (2022, methodology): “To that end, the interest in action understanding tasks such as recognition, anticipation, and temporal segmentation has grown, especially for egocentric views [5,17,34].”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “It is worth noting that Ego4D [24] is a video dataset, which consists of diverse ego-centric videos.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, methodology): “Alternatively, data of humans interacting in the world [14, 65, 16] is both diverse and contains useful interaction in scenes similar to those we would like robots to interact in.”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “There are typical real-world videos, which includes movies [51; 35], documentaries [35], TV series, egocentric videos [11], and surveillance footage [37].”
WorldSimBench: Towards Video Generation Models as World Simulators (2024, background): “…& Xie, 2023) and the extensive utilization of large-scale internet video datasets (Bain et al., 2021; Ebert et al., 2021; Goyal et al., 2017; Grauman et al., 2022), certain Predictive Actionable Video Model (Yang et al., 2023) models, also known as World Simulators, have achieved more…”
LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “Then, to enable more detailed temporal cues, we select several datasets commonly used in recent academic research for re-annotation, including Charades [124], Activi-tyNet [154], YouCook2 [171], and Ego4D [33].”