Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning

By Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Josh Tenenbaum, Chuang Gan. 2021

107 citations recorded by the source corpus.

We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, which are impractical to obtain in real life. In this paper, we present the Dynamic Concept Learner (DCL), a unified framework that grounds physical objects and events from video and language. DCL first adopts a trajectory extractor to track each object over time and to represent it as a latent, object-centric feature vector. Building upon this object-centric representation, DCL learns to approximate the dynamic interaction among objects using graph networks. DCL further incorporates a semantic parser to parse questions into semantic programs and, finally, a program executor to run the program to answer the question, levering the learned dynamics model. After training, DCL can detect and associate objects across the frames, ground visual properties, and physical events, understand the causal relationship between events, make future and counterfactual predictions, and leverage these extracted presentations for answering queries. DCL achieves state-of-the-art performance on CLEVRER, a challenging causal video reasoning dataset, even without using ground-truth attributes and collision labels from simulations for training. We further test DCL on a newly proposed video-retrieval and event localization dataset derived from CLEVRER, showing its strong generalization capacity.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Chuang Gan
3. Antonio Torralba
4. Jitendra Malik
5. Andrea Vedaldi
6. Xiaogang Wang
7. Dacheng Tao
8. Ping Luo

In Multimodal Learning

1. Chuang Gan
2. Josh Tenenbaum
3. Yining Hong
4. Zhenfang Chen
5. Antonio Torralba
6. Bo Wu
7. Shoubin Yu
8. Li Yi

Citations in the corpus

• STAR: A Benchmark for Situated Reasoning in Real-World Videos (2024, background): “Various benchmarks [11, 44, 20] have been constructed by asking annotators to provide designed questions [1, 44] or synthesizing questions with pre-defined templates [15, 24, 14, 6].”
• Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, methodology): “Both end-to-end [61, 19] and neuro-symbolic methods [77, 16] have been explored on CLEVRER.”
• PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning (2021, methodology): “Neural-symbolic methods [56, 42, 7, 38] explicitly perform symbolic reasoning on the objects representations and language representations.”
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning (2023, unknown): “The ideas of neural-symbolic reasoning models [12,46,65,66] offer a promising step-by-step transparent visual reasoning approach.”
• The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark Towards Physically Realistic Embodied AI (2021, unknown): “RL-Benchmark [21], Meta-World [51] and CLEVRER [50, 11]).”