Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

By Gunnar A. Sigurdsson, Abhinav Kumar Gupta, Cordelia Schmid, Ali Farhadi, Alahari Karteek. 2018

212 citations recorded by the source corpus.

In Actor and Observer we introduced a dataset linking the first and third-person video understanding domains, the Charades-Ego Dataset. In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68,536 activity instances in 68.8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available. Charades-Ego furthermore shares activity classes, scripts, and methodology with the Charades dataset, that consist of additional 82.3 hours of third-person video with 66,500 activity instances. Charades-Ego has temporal annotations and textual descriptions, making it suitable for egocentric video classification, localization, captioning, and new tasks utilizing the cross-modal nature of the data.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “We highlight SMs on egocentric perception because it is an important yet challenging computer vision domain (Grauman et al. 2021; Damen et al. 2020; Sigurdsson et al. 2018) with downstream applications in augmented reality (AR) and robotics (Ahn et al. 2022).”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, methodology): “…today’s largest egocentric datasets both in terms of hours of video (3,670 hours vs. 100 in [43]) and unique camera wearers (931 people vs. 71 in [201]); it spans hundreds of environments (rather than one or dozens, as in existing collections); and its video comes from 74 worldwide locations and…”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “The second is handobject interactions from [37], trained on 100K images from YouTube along with 42K images from three egocentric datasets [1,39,40] of which 18K are from our videos [1].”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Similarly good performing diverse architectures, all with higher-than-50% accuracy on Charades.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, methodology): “In [14], short segments linked to interactions with 30 daily objects are collected by querying YouTube, while [18, 42, 41] are scripted – subjects are requested to enact a crowd-sourced storyline [42, 41] or a given action [18], which oftentimes results in less natural looking actions.”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “Several egocentric video datasets have been created over the years, including [28, 29, 30].”
HARMONIC: A multimodal dataset of assistive human–robot collaboration (2018, background): “…et al. 2012), action recognition in third person video (Pirsiavash and Ramanan 2012; Sigurdsson et al. 2018a), relating first and third person videos as a proxy for theory of mind (Sigurdsson et al. 2018b), and learning about human social affordance from a third person view (Shu et al. 2016).”
Human Action Recognition and Prediction: A Survey (2018, unknown): “KTH [3] 2004 599 1 6 25 RGB Controlled Weizmann [71] 2005 90 1 10 9 RGB Controlled INRIA XMAS [199] 2006 390 5 13 10(3 times) RGB Controlled IXMAS [200] 2006 1,148 5 11 RGB Controlled UCF Sports [5] 2008 150 10 RGB Uncontrolled Hollywood [49] 2008 8 RGB Uncontrolled Hollywood2 [6] 2009 3,669 12 10 R”
Deep Multimodal Data Fusion (2024, background): “…[197], ScanRefer [21], ClevrTex [79] Video Captioning/Retrieval Video + Text Howto100M [122], Alivol-10M [89], YouCookII [242], Charades [160], TGIF [99], MSR-VTT [202], [6] Visual Question Answering Vision + Text video-text: TVQA [91], VQA 1.0 [7], VQA 2.0 [53], image-text: DAQUAR…”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “[10, 126, 145, 148] or third-person [9, 100, 159, 205, 207] videos; however, limited work considers both views together.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “Charades-ego [94]: A large-scale dataset of paired third and first person trimmed videos.”
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (2023, methodology): “Inspired by the success of image-text datasets (Sharma et al., 2018; Changpinyo et al., 2021), they used script-based programming (Schuldt et al., 2004; Kong et al., 2019; Sigurdsson et al., 2018) to extract millions of video-text data pairs.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While many datasets for thirdperson action recognition have been proposed throughout the years [30, 40, 72, 99], recently there is a surge in interest for data targeting also egocentric scenarios [12, 13, 29, 48, 62, 71] that involve mostly 2D features.”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “The ﬁeld is growing quickly in recent years, thanks in part to new ego-video benchmarks [6, 42, 55, 63].”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “Prior work on activity recognition has been mostly focused on recognizing short actions (Sigurdsson et al., 2018; Caba Heilbron et al., 2015; Fouhey et al., 2018), predicting pedestrian trajectories (Kitani et al.”
Egocentric Video-Language Pretraining (2022, background): “Several egocentric video datasets have been developed in decades, e.g., [20, 17, 31].”