Flexible Neural Representation for Physics Prediction

By Damian Mrowca, Chengxu Zhuang, E. Wang, Nick Haber, Li Fei-Fei, Josh Tenenbaum, Daniel L. K. Yamins. 2018

266 citations recorded by the source corpus.

Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail. Inspired by this ability, we propose a hierarchical particle-based object representation that covers a wide variety of types of three-dimensional objects, including both arbitrary rigid geometrical shapes and deformable materials. We then describe the Hierarchical Relation Network (HRN), an end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation. Compared to other neural network baselines, the HRN accurately handles complex collisions and nonrigid deformations, generating plausible dynamics predictions at long time scales in novel settings, and scaling to large scene configurations. These results demonstrate an architecture with the potential to form the basis of next-generation physics predictors for use in computer vision, robotics, and quantitative cognitive science.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “[7] Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Josh Tenenbaum, and Daniel L K Yamins.”
Toward Causal Representation Learning (2021, background): “Machine learning models that incorporate or learn structural knowledge of an environment have been shown to be more efficient and generalize better [9], [11], [15], [16], [27], [58], [77], [84], [85], [141], [157], [177], [181], [197], [211], [212], [244], [258], [272], [274].”
On the Binding Problem in Artificial Neural Networks (2020, background): “based on proximity (Chang et al., 2017; Mrowca et al., 2018) or a language parser (Tai et al.”
Learning to Simulate Complex Physics with Graph Networks (2020, background): “…to simulate rigid body, mass-spring, n-body, and robotic control systems (Battaglia et al., 2016; Chang et al., 2016; Sanchez-Gonzalez et al., 2018; Mrowca et al., 2018; Li et al., 2019; Sanchez-Gonzalez et al., 2019), as well as non-physical systems, such as multi-agent dynamics (Tacchetti et…”
Towards Sparse Hierarchical Graph Classifiers (2018, methodology): “…[28, 1], all earlier works in this area assume a ﬁxed, pre-deﬁned cluster assignment, that is obtained by running a clustering algorithm on the graph nodes, e.g. using the GraClus algorithm [7] to obtain structure-dependent cluster assignments or ﬁnding clusters via k-means on node features [21].”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, background): “[31]) may also ameliorate degeneracy and lead to more sophisticated behavior.”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, background): “…recent papers have explored the idea of incorporating physics priors into learning dynamics models of environments (Nguyen-Tuong and Peters, 2010; Cutler et al., 2014; Cutler and How, 2015; Scholz et al., 2014; Kansky et al., 2017; Battaglia et al., 2016; Mrowca et al., 2018; Xie et al., 2016).”
DiffTaichi: Differentiable Programming for Physical Simulation (2019, methodology): “Without differentiable programming, Battaglia et al. (2016), Chang et al. (2016) and Mrowca et al. (2018) used NNs to approximate the physical process and used the NN gradients as the approximate simulation gradients.”
Visual Grounding of Learned Physical Models (2020, background): “Mrowca et al. (2018) and Li et al. (2019a) extended the model to particle systems, which showed impressive results in simulating objects made of different materials like rigid objects and ﬂuids.”
Hamiltonian Graph Networks with ODE Integrators (2019, background): “1 Introduction Learning to simulate complex physical processes has been shown to benefit from rich inductive biases about the structure of the system’s state, such as modeling particles and their interactions via nodes and edges in a graph [1–5], and optimizing internal representations to predict kn”
Hierarchical Relational Inference (2020, background): “On the other hand, approaches that consider part-based representations of objects and hierarchical interaction graphs lack the capacity to learn from raw images and dynamically infer relationships (Mrowca et al. 2018; Lingelbach et al. 2020).”
Analogues of mental simulation and imagination in deep learning (2019, unknown): “Recently, models which represent the state as a graph [34 ] have been used to predict the motion of rigid bodies [35,36], deformable objects [37], articulated robotic systems [38], and multiagent systems [39].”
Learning Physical Graph Representations from Visual Scenes (2020, background): “Given the promising results of applying graph neural networks to dynamics prediction problems [36, 32, 33, 43], the structured latent state of PSGNet seems well-poised to connect visual perception with physical understanding.”
Graph Element Networks: adaptive, structured computation and memory (2019, background): “Recent approaches (Mrowca et al., 2018; Li et al., 2018) model objects with a node for each particle; requiring supervision for each of them.”
Learning to Control PDEs with Differentiable Physics (2020, methodology): “Lagrangian ﬂuid simulation has been tackled with regression forests (Ladicky et al., 2015), graph neural networks (Mrowca et al., 2018; Li et al., 2019), and continuous convolutions (Ummenhofer et al., 2020).”
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation (2020, background): “Training a Learnable Intuitive Physics Simulator The Hierarchical Relation Network (HRN) is a recently-published end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation [30].”
GraphCast: Learning skillful medium-range global weather forecasting (2022, methodology): “Our GraphCast architecture is most related to Keisler (2022)’s approach, and follows a long tradition of GNN-based learned simulators (Battaglia et al., 2016; Cranmer et al., 2020; Fortunato et al., 2022; Kipf et al., 2018; Li et al., 2018, 2019; Mrowca et al., 2018; Pfaff et al., 2021; Rubanova et ”