Visual Grounding of Learned Physical Models

By Yunzhu Li, Toru Lin, Kexin Yi, Daniel M. Bear, Daniel L. K. Yamins, Jiajun Wu, Josh Tenenbaum, Antonio Torralba. 2020

90 citations recorded by the source corpus.

Humans intuitively recognize objects' physical properties and predict their motion, even when the objects are engaged in complicated interactions. The abilities to perform physical reasoning and to adapt to new environments, while intrinsic to humans, remain challenging to state-of-the-art computational models. In this work, we present a neural model that simultaneously reasons about physics and makes future predictions based on visual and dynamics priors. The visual prior predicts a particle-based representation of the system from visual observations. An inference module operates on those particles, predicting and refining estimates of particle locations, object states, and physical parameters, subject to the constraints imposed by the dynamics prior, which we refer to as visual grounding. We demonstrate the effectiveness of our method in environments involving rigid objects, deformable materials, and fluids. Experiments show that our model can infer the physical properties within a few observations, which allows the model to quickly adapt to unseen scenarios and make accurate predictions into the future.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “, 2019), hierarchical physical reasoning (Mrowca et al., 2018; Li et al., 2020; Stanić et al., 2020), visual question answering (Santoro et al.”
Learning Physical Graph Representations from Visual Scenes (2020, background): “Given the promising results of applying graph neural networks to dynamics prediction problems [36, 32, 33, 43], the structured latent state of PSGNet seems well-poised to connect visual perception with physical understanding.”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, background): “Consistent with this, recurrent graph neural networks supervised on physical simulator states make extremely accurate predictions about full object trajectories [42, 37, 38, 53].”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “[22] reason about the underlying physical properties of objects that appear in a sequence of frames and predict future motion of those objects.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, unknown): “Physical Models Physical models are widely used in video prediction [48, 21, 74, 75], neural simulation and rendering [50, 52], and dynamic reasoning [10, 69].”