CLEVRER: CoLlision Events for Video REpresentation and Reasoning

By Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Josh Tenenbaum. 2019

645 citations recorded by the source corpus.

The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “Obtaining object-centric representations from raw perceptual input, such as an image or a video, is challenging and often requires either supervision [1, 3, 9, 10] or task-specific architectures [2, 11].”
STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Language Parser Language Parser parses each question to a functional program [18, 41] in the form of a program sentence.”
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions (2021, background): “More recently, CLEVRER [55] specially studied temporal and causal relationships of physical motions in simulated environments.”
Toward Causal Representation Learning (2021, background): “Machine learning models that incorporate or learn structural knowledge of an environment have been shown to be more efficient and generalize better [9], [11], [15], [16], [27], [58], [77], [84], [85], [141], [157], [177], [181], [197], [211], [212], [244], [258], [272], [274].”
COPHY: Counterfactual Learning of Physical Dynamics (2019, background): “CLEVRER (Yi et al., 2019) is a diagnostic video dataset for systematic evaluation of models on a wide range of reasoning tasks including counterfactual questions.”
Location-Aware Graph Convolutional Networks for Video Question Answering (2020, methodology): “Based on the visual sources, we can classify the VQA tasks into two categories: image QA (Goyal et al. 2017; Gan et al. 2017) and video QA (Lei et al. 2018; Yi et al. 2019).”
On Modality Bias in the TVQA Dataset (2020, neutral): “The CLEVRER dataset [48] aims to both reduce language bias by sampling evenly across question types and mitigate simpler pattern-recognition behaviour.”
DramaQA: Character-Centered Video Story Understanding with Hierarchical QA (2020, background): “Also, several datasets are proposed for temporal reasoning (Yi et al. 2020; Girdhar and Ramanan 2020), which shows intensive attraction to reasoning over the temporal structure.”
Long Context Transfer from Language to Vision (2024, background): “These range from fundamental visual perception tasks such as activity recognition[84], concept detection [80], and counting [28], to more complex visual reasoning tasks including compositional [24], causal [76, 83, 81], and situated reasoning [74].”
Cascade Reasoning Network for Text-based Visual Question Answering (2020, background): “Relational reasoning has been widely explored in computer vision tasks [15], from region classification [9] to visual question answering [8, 48, 49], and further promoted after graph neural networks appeared [36].”
AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning (2021, background): “Unfortunately, the most popular benchmarks and metrics defined to study compositional behavior have been limited to synthetic environments [25, 31, 21, 56] or to static images [17].”
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning (2021, background): “Several datasets [64, 16, 29, 49, 57] have been proposed to tackle this challenge.”
ShareGPT4Video: Improving Video Understanding and Generation with Better Captions (2024, neutral): “We provide some statistics on generated captions in Figure 8.”
EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding (2023, background): “…[15], complex web video understanding [63], situated reasoning [50], spatiotemporal reasoning [28], social intelligence [65], dynamic neuro-symbolic reasoning [62], external knowledge-based reasoning [16] and many more [37, 61, 42, 10, 9, 13, 45, 56, 30, 31, 8, 64, 11, 32, 59, 66, 52, 24].”
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning (2019, background): “Concurrent to us, CLEVRER (Yi et al., 2020), PHYRE (Bakhtin et al., 2019), COPHY (Baradel et al., 2020) and IntPhys (Riochet et al., 2018) benchmarks have been proposed with a focus on causal physical reasoning through QA, RL, prediction and ranking interfaces respectively.”
Visual Grounding of Learned Physical Models (2020, background): “Other researchers have shown better performance with intermediate representations like instance masks (Fragkiadaki et al., 2016; Watters et al., 2017; Janner et al., 2019; Yi et al., 2020), object keypoints (Minderer et al.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, methodology): “Both end-to-end [61, 19] and neuro-symbolic methods [77, 16] have been explored on CLEVRER.”
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation (2020, background): “Taken together, TDW is substantially more full-featured for supporting future development in rapidly-expanding research areas such as learning scene dynamics for physical reasoning [52, 51] and model-predictive planning and control [4, 32, 16, 5, 10, 2, 39, 15, 35].”