STAR: A Benchmark for Situated Reasoning in Real-World Videos

By Bo Wu, Shoubin Yu. 2024

322 citations recorded by the source corpus.

Reasoning in the real world is not divorced from situations. How to capture the present knowledge from surrounding situations and perform reasoning accordingly is crucial and challenging for machine intelligence. This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR Benchmark). This benchmark is built upon the real-world videos associated with human actions or interactions, which are naturally dynamic, compositional, and logical. The dataset includes four types of questions, including interaction, sequence, prediction, and feasibility. We represent the situations in real-world videos by hyper-graphs connecting extracted atomic entities and relations (e.g., actions, persons, objects, and relationships). Besides visual perception, situated reasoning also requires structured situation comprehension and logical reasoning. Questions and answers are procedurally generated. The answering logic of each question is represented by a functional program based on a situation hyper-graph. We compare various existing video reasoning models and find that they all struggle on this challenging situated reasoning task. We further propose a diagnostic neuro-symbolic model that can disentangle visual perception, situation abstraction, language understanding, and functional reasoning to understand the challenges of this benchmark.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “We show our zero-shot STAR results in Table 4 in the main text.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “[49] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan.”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, unknown): “V TT [ 199 ] M S R V TT - QA [ 198 ] M S VD - QA [ 198 ] S T A R [ 196 ] S EE D -”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, unknown): “909 between human and LLM evaluation (bootstrap CI=(0.883,0.928), N=9999), indicating excellent agreement with human judgement.”
MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding (2024, unknown): “There are specialized benchmarks for temporal perception [49; 42], action understanding [42; 41], video classification [14], video reasoning [45; 44], and video captioning [47; 30].”
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (2023, methodology): “To balance the question difficulty, we design the selection criteria for STAR [82] and CLEVRER [30].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…WebSight [122] Type: Video Datasets Captioning Mementos [254], ShareGPT4Video [30], VideoGPT+ [174], ShareGPT4o-Video [35] General QA VideoChat2-IT [131], EgoTaskQA [99], NTU RGB+D [152], CLEVRER [276], STAR [259], LSMDC [201] Table 4: Summary of the pre-training data mixture of InternVL 2.5.”
Long Context Transfer from Language to Vision (2024, background): “These range from fundamental visual perception tasks such as activity recognition[84], concept detection [80], and counting [28], to more complex visual reasoning tasks including compositional [24], causal [76, 83, 81], and situated reasoning [74].”
Balanced Multimodal Learning via On-the-fly Gradient Modulation (2022, background): “timodal learning has exhibited a clear advantage in improving the performance of previous uni-modal tasks as well as addressing new challenging problems, such as action recognition [10,23,29,37], audio-visual speech recognition [33], and visual question answering [2, 19, 41].”
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (2024, methodology): “We also include NExT-QA [57] and STAR [55] to expand our video training data.”