CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning

By Rohit Girdhar, Deva Ramanan. 2019

209 citations recorded by the source corpus.

Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets. However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements. Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive. We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure. In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved. Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning. In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable. Using CATER, we provide insights into some of the most recent state of the art deep video architectures.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “CATER [10] asks models to recognize objects and actions in synthetic videos, requiring an understanding of spatio-temporal compositions of object movements.”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “CATER (Girdhar & Ramanan, 2020) introduces a synthetic video dataset for temporal reasoning associated with compositional actions.”
Coordination Among Neural Modules Through a Shared Global Workspace (2021, background): “Cater is a spatio-temporal reasoning video dataset introduced in Girdhar and Ramanan (2019).”
Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations (2020, background): “Concerning our data set, an interesting next step would be to create a confounded causal data set in the approach of [10].”
TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “…that video understanding models may utilize language priors or single-frame bias as shortcuts to obtain the correct answer, without truly understanding the temporal content of a video (Huang et al., 2018; Buch et al., 2022; Sevilla-Lara et al., 2019; Lei et al., 2022; Girdhar and Ramanan, 2019).”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, methodology): “Many video question answering datasets [25, 47, 40, 65, 79] and the methods [80, 49, 36, 73, 76, 20] built on them mainly focus on understanding diverse visual scenes, such as human actions (MovieQA [65]) or 3D object movements without physical and language cues (CATER [25]).”
On Modality Bias in the TVQA Dataset (2020, unknown): “Girdhar and Ramanan [18] create the CATER dataset using a homogeneous scene and object set with multiple variable dataset parameters to ‘fully control’ object and scene bias.”
Revisiting the “Video” in Video-Language Understanding (2022, background): “Standard video-language tasks include both discriminative tasks, such as video question answering [12, 19, 27, 28, 30, 54, 56, 58, 59], text-to-video/moment retrieval [16,23,43,56,62], and generative tasks, such as video cap-tioning [4, 23] and open-ended VQA [54, 56].”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “Notably, they suﬀer from a major drawback [17]; activities are highly correlated to the general scene and object context, possessing a strong dataset bias for activity understanding.”
Conditional Object-Centric Learning from Video (2021, methodology): “The original CATER (Girdhar & Ramanan, 2019) dataset (without segmentation mask annotations) is publicly available under an Apache 2.0 license.”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, methodology): “Our benchmark draws inspiration from diagnostic synthetic datasets like CATER [25] or CLEVRER [54], behavioral tests like the Visual Turing Test [41, 23], experiments in developmental psychology [1, 6, 31], and motor-free perception screening tests used for children or adults [42, 22].”
DramaQA: Character-Centered Video Story Understanding with Hierarchical QA (2020, background): “Also, several datasets are proposed for temporal reasoning (Yi et al. 2020; Girdhar and Ramanan 2020), which shows intensive attraction to reasoning over the temporal structure.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, background): “Recently, various benchmarks (Riochet et al., 2018; Bakhtin et al., 2019; Girdhar & Ramanan, 2020; Baradel et al., 2020; Gan et al., 2020) have been proposed to study dynamics and reasoning in physical scenes.”
Anticipative Video Transformer (2021, background): “Moreover, while action recognition can often side-step temporal reasoning by leveraging instantaneous contextual cues [31], anticipation inherently requires modeling the progression of past actions to predict the future.”
SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments (2021, background): “But with the advancement and rise of deep learning, computer graphics, and embodied AI [8, 15, 18], there has been a paradigm shift towards generating synthetic datasets that range from simple 2D cartoons [10, 22, 35] to realistic interaction in 3D environments [3, 7, 9, 13, 29, 34], all with the ai”
Revealing Occlusions with 4D Neural Fields (2022, methodology): “We extend CATER [18] (which is in turn based on CLEVR [23]) mainly in order to increase the degree of occlusions, and call our proposed dataset GREATER.”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “In contrast, the other category of AI approaches [12, 13, 14, 15] focuses primarily on heuristic rule-based causal reasoning [16, 17, 2].”