COPHY: Counterfactual Learning of Physical Dynamics

By Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, Christian Wolf. 2019

114 citations recorded by the source corpus.

Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Causal Inference & Discovery, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Causal Inference & Discovery

In AI for Physical Sciences

Citations in the corpus

CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “Moreover, causality has been historically studied from the perspective of probabilistic and causal reasoning (Pearl, 2009), cognitive psychology (Griffiths & Tenenbaum, 2005), and more recently in the context of machine learning (Goyal et al., 2019b; Schölkopf, 2019; Baradel et al., 2019; Bakhtin et”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “Concurrent to our work, CoPhy (Baradel et al., 2020) studies physical dynamics prediction in a counterfatual setting.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, methodology): “In some datasets, such pairs are provided explicitly [7,12,47,48].”
SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments (2021, background): “But with the advancement and rise of deep learning, computer graphics, and embodied AI [8, 15, 18], there has been a paradigm shift towards generating synthetic datasets that range from simple 2D cartoons [10, 22, 35] to realistic interaction in 3D environments [3, 7, 9, 13, 29, 34], all with the ai”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “There exist a number of video prediction [Yi et al., 2019, Baradel et al., 2019] and visual question answering [Johnson et al., 2017] datasets that also make use of a blocks world for visual representation.”
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning (2019, background): “Concurrent to us, CLEVRER (Yi et al., 2020), PHYRE (Bakhtin et al., 2019), COPHY (Baradel et al., 2020) and IntPhys (Riochet et al., 2018) benchmarks have been proposed with a focus on causal physical reasoning through QA, RL, prediction and ranking interfaces respectively.”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, background): “Recently, various benchmarks (Riochet et al., 2018; Bakhtin et al., 2019; Girdhar & Ramanan, 2020; Baradel et al., 2020; Gan et al., 2020) have been proposed to study dynamics and reasoning in physical scenes.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, methodology): “Datasets and Evaluation Settings To validate the effectiveness of our method for reasoning about the physical world, we conduct main experiments on the CLEVRER [77] dataset, as it contains both language and physics cues such as rigid body collisions and dynamics, compared to other benchmarks that fo”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “…range from physics-based prediction models using pixel-level convolutional networks [31, 32, 33, 34, 29, 35] and scene-based learning [14, 36, 37, 38] to learning of physical representations of the intuitive physics [6, 2, 39] behind the interactions to make accurate prediction of the outcomes…”
Better Set Representations For Relational Reasoning (2020, methodology): “First, the Baseline model implements G using a CNN with groups of channels of the ﬁnal feature map interpreted as set elements; this follows the approach of differentiable object extractors [1, 10].”