Visual Interaction Networks: Learning a Physics Simulator from Video

By Nicholas Watters, Daniel Zoran, Theophane Weber, Peter W. Battaglia, Razvan Pascanu, Andrea Tacchetti. 2017

297 citations recorded by the source corpus.

From just a glance, humans can make rich predictions about the future state of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains and require direct measurements of the underlying states. We introduce the Visual Interaction Network, a general-purpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions and dynamics, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. Our results demonstrate that the perceptual module and the object-based dynamics predictor module can induce factored latent representations that support accurate dynamical predictions. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Object-Centric Learning with Slot Attention (2020, background): “Obtaining object-centric representations from raw perceptual input, such as an image or a video, is challenging and often requires either supervision [1, 3, 9, 10] or task-specific architectures [2, 11].”
Relation Networks for Object Detection (2017, methodology): “Attention modules in NLP and physical system modeling Attention modules have recently been successfully applied in the NLP field [20, 21, 6, 56] and in physical system modeling [2, 57, 31, 45, 13, 42].”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, neutral): “Particularly, we believe that exploring attention (Vaswani et al., 2017; Bahdanau et al., 2014) and relational inductive biases (Watters et al., 2017; Battaglia et al., 2016) could be useful in visual domains like Atari2600.”
FiLM: Visual Reasoning with a General Conditioning Layer (2017, methodology): “n Figure 3. We turn the parameters of batch normalization layers that immediately precede FiLM layers off. Drawing from prior work on CLEVR (Hu et al. 2017; Santoro et al. 2017) and visual reasoning (Watters et al. 2017), we concatenate two coordinate feature maps indicating relative x and y spatial”
E(n) Equivariant Graph Neural Networks (2021, background): “, 2018), and physical systems simulations (Grzeszczuk et al., 1998; Watters et al., 2017).”
Towards Causal Representation Learning (2021, background): “among them, modulo the completeness of the variables, and confounding [58, 265, 43, 264, 14, 136].”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “…al., 2016; Shao et al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from images (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017; Lerer et al., 2016; Mottaghi et al., 2016; Fragkiadaki et al., 2016), or from a symbolic, abstract representation of the…”
Recurrent Independent Mechanisms (2019, background): “…like physics simulation, multi-object scenes, and motion-capture data, and multiagent systems (Scarselli et al., 2008; Bronstein et al., 2017; Watters et al., 2017; Raposo et al., 2017; Santoro et al., 2017; Gilmer et al., 2017; Van Steenkiste et al., 2018; Kipf et al., 2018; Battaglia et…”
Structured agents for physical construction (2019, methodology): “Within the domain of physical reasoning, graph neural networks have been used as forward models for predicting future states and images (Battaglia et al., 2016; Chang et al., 2017; Watters et al., 2017; van Steenkiste et al., 2018), and can allow efficient learning and rich generalization.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “Interaction Networks (Battaglia et al., 2016; Watters et al., 2017) and the Neural Physics Engine Chang et al. (2017) use a full GN but for the absence of the global to update the edge properties: That work also included an extension to the above formulation which output global, rather than…”
Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow (2018, background): “l for giving NNs predictive capabilities for complex inverse problems. E.g., giving models the ability to understand, or at least have an intuition about physics, could help us to analyze video data [Watters et al. 2017] or even control robots [Schenck and Fox 2017]. The complexity of nature at huma”
Meta-learning autoencoders for few-shot prediction (2018, background): “This class of problems is important in many areas, for example, learning and predicting physics [8, 26, 25, 3], reinforcement learning of games [15, 10], unsupervised learning of videos [20], and applications such as self-driving, where we cannot enumerate and train with all environments that the…”
Graph networks as learnable physics engines for inference and control (2018, neutral): “…et al., 2013; Li et al., 2015; Henaff et al., 2015; Du-venaud et al., 2015; Dai et al., 2016; Defferrard et al., 2016; Niepert et al., 2016; Kipf & Welling, 2016; Battaglia et al., 2016; Watters et al., 2017; Raposo et al., 2017; Santoro et al., 2017; Bronstein et al., 2017; Gilmer et al., 2017).”
Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs (2019, methodology): “However, the CoordConv technique had been used beforehand [Zhao et al., 2015, Liang et al., 2015, Watters et al., 2017, Wojna et al., 2017, Perez et al., 2017, Nash et al., 2017, Ulyanov et al., 2017] and its origin is unclear.”
Linearly-Recurrent Autoencoder Networks for Learning Dynamics (2017, unknown): “The underlying philosophy of our approach is similar to Visual Interaction Networks (VINs) [32] that learn physics-based dynamics models for encoded latent variables.”
Hamiltonian Neural Networks (2019, background): “Others, such as Interaction Networks or Relational Networks were meant to be fully general [43, 34, 2].”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “[36] also infers the physical state of objects from video input and predict their future trajectories.”