Taking Visual Motion Prediction To New Heightfields

By Sébastien Ehrhardt, Áron Monszpart, Niloy J. Mitra, Andrea Vedaldi. 2017

24 citations recorded by the source corpus.

While the basic laws of Newtonian mechanics are well understood, explaining a physical scenario still requires manually modeling the problem with suitable equations and estimating the associated parameters. In order to be able to leverage the approximation capabilities of artificial intelligence techniques in such physics related contexts, researchers have handcrafted the relevant states, and then used neural networks to learn the state transitions using simulation runs as training data. Unfortunately, such approaches are unsuited for modeling complex real-world scenarios, where manually authoring relevant state spaces tend to be tedious and challenging. In this work, we investigate if neural networks can implicitly learn physical states of real-world mechanical processes only based on visual data while internally modeling non-homogeneous environment and in the process enable long-term physical extrapolation. We develop a recurrent neural network architecture for this task and also characterize resultant uncertainties in the form of evolving variance estimates. We evaluate our setup to extrapolate motion of rolling ball(s) on bowls of varying shape and orientation, and on arbitrary heightfields using only images as input. We report significant improvements over existing image-based methods both in terms of accuracy of predictions and complexity of scenarios; and report competitive performance with approaches that, unlike us, assume access to internal physical states.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Economics and Econometrics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Economics and Econometrics

Citations in the corpus

Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “Ehrhardt et al. (2017) proposed to predict a Gaussian distribution for the uncertain future states of a sphere.”
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning (2018, methodology): “Comparing with representative video prediction (Lee et al., 2018; Babaeizadeh et al., 2018) and physical prediction (Ehrhardt et al., 2017; Mottaghi et al., 2016; Li et al., 2017; Lerer et al., 2016) methods, our approach achieves substantially better results at tasks that require building…”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “First, we consider a relatively simple dataset, BALLSINBOWL from [6], for assessing the model features and ablations.”
Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids (2018, background): “Researchers have developed many differentiable physical simulators (Ehrhardt et al., 2017; Degrave et al., 2016; Todorov et al., 2012), among which some can also provide analytical gradients (Tedrake & the Drake Development Team, 2019).”