Interaction Networks for Learning about Objects, Relations and Physics

By Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, Koray Kavukcuoglu. 2016

1,568 citations recorded by the source corpus.

Reasoning about objects, relations, and physics is central to human intelligence, and a key goal of artificial intelligence. Here we introduce the interaction network, a model which can reason about how objects in complex systems interact, supporting dynamical predictions, as well as inferences about the abstract properties of the system. Our model takes graphs as input, performs object- and relation-centric reasoning in a way that is analogous to a simulation, and is implemented using deep neural networks. We evaluate its ability to reason about several challenging physical domains: n-body problems, rigid-body collision, and non-rigid dynamics. Our results show it can be trained to accurately simulate the physical trajectories of dozens of objects over thousands of time steps, estimate abstract quantities such as energy, and generalize automatically to systems with different numbers and configurations of objects and relations. Our interaction network implementation is the first general-purpose, learnable physics engine, and a powerful general framework for reasoning about object and relations in a wide variety of complex real-world domains.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “Instead, models are typically trained to operate on a structured representation of the environment that is obtained, for example, from the internal representation of a simulator [6, 8] or of a game engine [4, 5].”
Relation Networks for Object Detection (2017, methodology): “fective, for the ﬁrst time. Attention modules in NLP and physical system modelingAttention modules have recently been successfully applied in the NLP ﬁeld [20,21,6,56] and in physical system modeling [2,57,31,45,13,42]. The attention module can well capture the long-term dependencies in these proble”
Non-local Neural Networks (2017, background): “Interaction Networks (IN) [2, 52] were proposed recently for modeling physical systems.”
What can the brain teach us about building artificial intelligence? (2016, background): “ssumptions about compositional and causal structure, yielding architectures that learn efficiently about the dynamics of physical systems, and even generalize to previously unseen numbers of objects (Battaglia et al., 2016), another challenge problem highlighted by Lake and colleagues. In such cases”
Programmable Agents (2017, methodology): “Our message passing operation closely resembles an Interaction Network [4] with some additional features.”
VAIN: Attentional Multi-agent Predictive Modeling (2017, positive): “Two related approaches that learn multi-agent interactions on a graph structure are: Interaction Networks [9] which learn a physical simulation of objects that exhibit binary relations and Communication Networks (CommNets) [11], presented for learning optimal communications between agents.”
A simple neural network module for relational reasoning (2017, background): “Although several other models supporting relation-centric computation have been proposed, such as Graph Neural Neworks, Gated Graph Sequence Neural Netoworks, and Interaction Networks, [20, 13, 2], RNs are simpler, more exclusively focused on general relation reasoning, and easier to integrate withi”
One-Shot Imitation Learning (2017, background): “The interaction networks proposed in [7, 12] also leverage locality of physical interaction in learning.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Some modeling choices include using pixels [28, 100], graphs [10], euclidean distance using sensors [2], or studying cycle consistency across time Epstein et al.”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Particularly, we believe that exploring attention (Vaswani et al., 2017; Bahdanau et al., 2014) and relational inductive biases (Watters et al., 2017; Battaglia et al., 2016) could be useful in visual domains like Atari2600.”
Neural Message Passing for Quantum Chemistry (2017, methodology): “Deep Tensor Neural Networks, Schütt et al. (2017) The message from w to v is computed by Mt = tanh ( W fc((W cfhtw + b1) (W dfevw + b2)) ) where W fc, W cf , W df are matrices and b1, b2 are bias vectors.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “, 2018), and a range of works on predicting physical phenomena have utilized offline datasets (Lerer et al., 2016; Battaglia et al., 2016).”
Towards Causal Representation Learning (2021, background): “among them, modulo the completeness of the variables, and confounding [58, 265, 43, 264, 14, 136].”
On the Binding Problem in Artificial Neural Networks (2020, background): “MPNNs have been shown to generalize more systematically (compared to standard neural networks) on many different tasks that require relational responding in terms of objects, including common-sense physical reasoning (Chang et al., 2017; Battaglia et al., 2016; Janner et al., 2019), hierarchical phy”
Learning to Simulate Complex Physics with Graph Networks (2020, background): “GNs and their variants, e.g., “interaction networks”, can learn to simulate rigid body, mass-spring, n-body, and robotic control systems (Battaglia et al., 2016; Chang et al., 2016; Sanchez-Gonzalez et al., 2018; Mrowca et al., 2018; Li et al., 2019; Sanchez-Gonzalez et al., 2019), as well as…”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, methodology): “We apply the Propagation Network (PropNet) (Li et al., 2019) for dynamics modeling, which is a learnable physics engine that, extending from the interaction networks (Battaglia et al., 2016), performs object-and relation-centric updates on a dynamic scene.”
Structured agents for physical construction (2019, methodology): “Within the domain of physical reasoning, graph neural networks have been used as forward models for predicting future states and images (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017; van Steenkiste et al., 2018), and can allow efficient learning and rich generalization.”