EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

By Karttikeya Mangalam, Raiymbek Akshulakov, Jitendra Malik. Neural Information Processing Systems, 2023

738 citations recorded by the source corpus.

We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks&datasets. Based on this metric, we find EgoSchema to have intrinsic temporal lengths over 5.7x longer than the second closest dataset and 10x to 100x longer than any other video understanding dataset. Further, our evaluation of several current state-of-the-art video and language models shows them to be severely lacking in long-term video understanding capabilities. Even models with several billions of parameters achieve QA accuracy less than 33% (random is 20%) on the EgoSchema multi-choice question answering task, while humans achieve about 76% accuracy. We posit that \name{}{}, with its long intrinsic temporal structures and diverse complexity, would serve as a valuable evaluation probe for developing effective long-term video understanding systems in the future. Data and Zero-shot model evaluation code are open-sourced for both public and commercial use under the Ego4D license at http://egoschema.github.io

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/656a6b3c0348d69cf9f98f95cbf68046941a4f29

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Limin Wang
3. Ping Luo
4. Yali Wang
5. Kunchang Li
6. Yi Wang
7. Yinan He
8. Jiashuo Yu

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Luc Van Gool
3. Ping Luo
4. Dacheng Tao
5. Limin Wang
6. Ming-Hsuan Yang
7. Xiaogang Wang
8. Trevor Darrell

Citations in the corpus

• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding (2024, methodology): “When applying uniforming sampling, 144 tokens per frame shows better performance than 64 tokens in an 8k context length on VideoMME (Fu et al., 2024) and MLVU Zhou et al. (2024) while worse on EgoSchema Mangalam et al. (2024).”
• InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, background): “…- - VAST [Chen et al., 2024b AST [Gong et al., 2021] 95.60 BEATs [Chen et al., 2023a] 98.10 InternVideo2 s 2 98.60 Table 14: Results of Chat-centric Evaluation on Multiple Choice Video-QA on MVBench [Li et al., 2023d], Egoschema [Mangalam et al., 2023] and Perception Test [Patraucean et al., 2023].”
• MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, neutral): “Beyond movies, there are other task-specific benchmarks like EgoSchema [29], which presents video reasoning tasks using first-person footage from Ego4D [11].”
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (2024, background): “Existing LMM benchmarks for long videos typically focus on a specific category of videos, e.g. egocentric videos [Mangalam et al., 2023], or movies [Song et al., 2023, Zhang et al., 2023a].”
• Kimi-VL Technical Report (2025, neutral): “EgoSchema (Mangalam et al. 2023) is a video benchmark designed to evaluate the long-form video understanding capabilities within the ego-centric scenario.”
• MovieChat: From Dense Token to Sparse Memory for Long Video Understanding (2023, background): “[63] successfully builds a benchmark contains multiple sources of information ( e.g ., video clips, plots, and DVS) for question-answering tasks in the movie field.”
• Video Instruction Tuning With Synthetic Data (2024, neutral): “Additionally, integrating high-quality image data provided modest benefits on EgoSchema.”
• VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, neutral): “We assess VideoLLaMA3’s general video understanding capabilities through established benchmarks: (1) Multi-Choice Video Question Answering (MC-VQA) tasks, including MVBench [26], VideoMME [126], EgoSchema [127], and Perception-Test [128].”
• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “This upscaling results in notable improvements across multiple benchmarks, with VideoLLaMA 2-8 × 7B achieving the highest accuracies on Egoschema, Perception-Test, and Video-MME.”
• VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, neutral): “EgoSchema [28].”
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “While Large Language Models (LLMs) [3, 6, 9, 36, 61, 67, 68, 77, 81, 82, 88, 105] have advanced linguistic intelligence, visual-spatial intelligence remains under-explored, despite its relevance to robotics [7, 8, 21, 64], autonomous driving [79], and AR/VR [12, 27, 55].”
• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (2024, neutral): “However, existing video-based benchmarks [27, 32, 41, 45] are still limited to thoroughly reveal their performance, such as a lack of diversity in video types, insufficient coverage of temporal dynamics, and the narrow focus on a single modality.”
• LongVILA: Scaling Long-Context Visual Language Models for Long Videos (2024, neutral): “In contrast, the LongVILA model (right) demonstrated enhanced performance across a range of frame counts and depths.”
• LLaVA-OneVision: Easy Visual Task Transfer (2024, neutral): “These include ActivityNet-QA [154] that contains human-annotated action-related QA pairs derived from ActivityNet dataset, EgoSchema [98] and MLVU [169] focusing on long video understanding, PerceptionTest [115] designed to evaluate the perception skills, VideoMME [29] and NeXTQA [141] containing…”
• LLaVA-Video: Video Instruction Tuning With Synthetic Data (2024, neutral): “However, this impact is less pronounced in larger models, as demonstrated by the LLaVA-Video-72B model’s superior performance over LLaVA-OV-72B in EgoSchema.”
• Qwen2.5-Omni Technical Report (2025, neutral): “We assess our model on several representative video understanding tasks like Video-MME (Fu et al., 2024a), MVBench (Li et al., 2024a), and EgoSchema (Mangalam et al., 2023).”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…(en) [276], LLaVA-Video (en) [307], FineVideo (en) [67], PerceptionTest (en) [193], HiREST (en) [291], STAR (en) [259], General QA EgoSchema (en) [175], ScanQA (en) [10], LSMDC (en) [201] GUI GUI-World (en) [24] Type: Text Datasets UltraFeedback (en) [48], UltraChat (en) [58],…”
• LVBench: An Extreme Long Video Understanding Benchmark (2024, neutral): “…NExT-QA [41] 52,044 44 Manual How2QA [18] 44,007 60 Manual ActivityNet-QA [45] 800 111 Manual CinePile [31] 303,828 160 Auto&Manual EgoSchema [24] 5,000 180 Auto&Manual MovieQA [36] 6,462 203 Manual LongVideoBench [40] 6,678 473 Manual MovieChat-1K [34] 13,000 564 Manual MoVQA [49] 21,953…”
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, background): “More results on NExT-QA [84], STAR [82], TVQA [35], EgoSchema [52] and IntentQA [40] can be found in the appendix.”