A Compositional Object-Based Approach to Learning Physical Dynamics

By Michael Chang, Tomer D. Ullman, Antonio Torralba, Josh Tenenbaum. 2016

470 citations recorded by the source corpus.

We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

Citations in the corpus

Programmable Agents (2017, methodology): “The second recent trend we follow is the rise of relational neural network architectures, particularly Interaction Networks [4] and the Neural Physics Engine [8].”
One-Shot Imitation Learning (2017, background): “The interaction networks proposed in [7, 12] also leverage locality of physical interaction in learning.”
Interaction Networks for Learning about Objects, Relations and Physics (2016, methodology): “A similar approach has been developed independently by Chang et al. [4].”
Toward Causal Representation Learning (2021, background): “Structured generative approaches further aim at decomposing an environment into multiple entities with causally correct relations among them, modulo the completeness of the variables, and confounding [15], [44], [59], [136], [264], [265].”
Reconciling deep learning with symbolic artificial intelligence: representing objects and relations (2019, background): “Simulation, in the sense of using a model to predict how states unfold over time, is one form of such processing, and a number of recent works describe deep neural networks that carry out such simulations [46,47].”
Towards Causal Representation Learning (2021, background): “among them, modulo the completeness of the variables, and confounding [58, 265, 43, 264, 14, 136].”
On the Binding Problem in Artificial Neural Networks (2020, background): “MPNNs have been shown to generalize more systematically (compared to standard neural networks) on many different tasks that require relational responding in terms of objects, including common-sense physical reasoning (Chang et al., 2017; Battaglia et al., 2016; Janner et al., 2019), hierarchical phy”
Learning to Simulate Complex Physics with Graph Networks (2020, background): “…variants, e.g., “interaction networks”, can learn to simulate rigid body, mass-spring, n-body, and robotic control systems (Battaglia et al., 2016; Chang et al., 2016; Sanchez-Gonzalez et al., 2018; Mrowca et al., 2018; Li et al., 2019; Sanchez-Gonzalez et al., 2019), as well as non-physical…”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “…physical and causal reasoning (Lerer et al., 2016; Battaglia et al., 2013; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Battaglia et al., 2016; Chang et al., 2017; Agrawal et al., 2016; Finn et al., 2016; Shao et al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from…”
Structured agents for physical construction (2019, methodology): “Within the domain of physical reasoning, graph neural networks have been used as forward models for predicting future states and images (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017; van Steenkiste et al., 2018), and can allow efficient learning and rich generalization.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “Interaction Networks (Battaglia et al., 2016; Watters et al., 2017) and the Neural Physics Engine Chang et al. (2017) use a full GN but for the absence of the global to update the edge properties: That work also included an extension to the above formulation which output global, rather than…”
Flexible Neural Representation for Physics Prediction (2018, background): “Our work draws inspiration from and extends that of Chang et al. [11] and Battaglia et al. [4], which in turn use ideas from graph-based neural networks [39, 44, 9, 30, 22, 14, 13, 24, 8, 40].”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, methodology): “They maintain explicit object representations, which allow for general physical dynamics to be learned between object pairs in a compositional manner (Battaglia et al., 2016; Chang et al., 2016; Watters et al., 2017).”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, background): “Chang et al. [7] introduced a similar model in parallel that could likewise predict rigid body dynamics.”
Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks (2019, background): “Indeed, recent work in machine learning is replete with examples of the beneﬁts of object-like representations (Diuk, Cohen, and Littman 2008; Chang et al. 2016; Kansky et al. 2017; Santoro et al. 2017).”
Differentiable Physics-informed Graph Networks (2019, background): “Learning physical dynamics A class of models (Grzeszczuk et al., 1998; Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017; Sanchez-Gonzalez et al., 2018; Kipf et al., 2018) have been proposed based on the assumption that neural networks can learn complex physical interactions and…”
Few-Shot Learning with Graph Neural Networks (2017, positive): “One-shot learning was first introduced by Fei-Fei et al. (2006), they assumed that currently learned classes can help to make predictions on new ones when just one or few labels are available. More recently, Lake et al. (2015) presented a Hierarchical Bayesian model that reached human level error on”