Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images

By Roozbeh Mottaghi, Hessam Bagherinezhad, Mohammad Rastegari, Ali Farhadi. 2015

157 citations recorded by the source corpus.

In this paper, we study the challenging problem of predicting the dynamics of objects in static images. Given a query object in an image, our goal is to provide a physical understanding of the object in terms of the forces acting upon it and its long term motion as response to those forces. Direct and explicit estimation of the forces and the motion of objects from a single image is extremely challenging. We define intermediate physical abstractions called Newtonian scenarios and introduce Newtonian Neural Network (N3) that learns to map a single image to a state in a Newtonian scenario. Our evaluations show that our method can reliably predict dynamics of a query object from a single image. In addition, our approach can provide physical reasoning that supports the predicted dynamics in terms of velocity and force vectors. To spur research in this direction we compiled Visual Newtonian Dynamics (VIND) dataset that includes more than 6000 videos aligned with Newtonian scenarios represented using game engines, and more than 4500 still images with their ground truth dynamics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Structured agents for physical construction (2019, background): “…approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; Groth et al., 2018;…”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “( Vondrick et al., 2016a) train a conditional CNN to predict future feature vectors, and several papers (e.g., (Lerer et al., 2016; Mottaghi et al., 2016; Wu et al., 2017; van Steenkiste et al., 2018; Fragkiadaki et al., 2016; Watters et al., 2017)) predict future object states.”
Flexible Neural Representation for Physics Prediction (2018, background): “However, future predictors operating directly on 2D image representations often fail to generate sharp object boundaries and struggle with occlusions and remembering objects when they are no longer visually observable [1, 17, 16, 28, 29, 33, 34, 19].”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, background): “[32, 23, 14] proposed using Newtonian physics in combination with neural networks to predict the dynamics of objects in the future.”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “Learning physics: Several works have explicitly addressed prediction of physical interactions, including predicting ball motion [5], block falling [2], the effects of forces [19, 18], future human interactions [9], and future car trajectories [26].”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, background): “Recently, physics engines have been used to learn the dynamics of real-world scenes from images [36], [37], [38].”
Learning Visual Predictive Models of Physics for Playing Billiards (2015, methodology): “The early development of smooth pursuit suggests that it is important for further development of visual and motor faculties of a human infant.”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, background): “Others predict labeled human walking trajectories [14] or object trajectories [15] in restricted domains.”
IQA: Visual Question Answering in Interactive Environments (2017, background): “isual Learning by Simulation: There has been an increased use of simulated environments and game platforms to train computer vision systems to perform tasks such as learning the dynamics of the world [47,48,70], semantic segmentation [18], pedestrian detection [43], pose estimation [53] and urban dr”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, background): “Similar to [24], the input to our image tower is a four-channel RGB-M image, where we add a mask channel (M) to the RGB image.”
Physics 101: Learning Physical Object Properties from Unlabeled Videos (2016, background): “Recently, researchers started to tackle concrete scenarios for understanding physics from vision [11, 15, 18, 19, 23, 24, 25, 27], some building on recent advances in deep learning.”
SE3-nets: Learning rigid body motion using deep neural networks (2016, background): “Physics prediction: Recent work in deep learning has looked at the problem of predicting the stability of stacked blocks [15], ball motion [16], and object dynamics in images [17].”
To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction (2016, neutral): “[21] aims at understanding dynamic events governed by laws of Newtonian physics, but uses proto-typical motion scenarios as exemplars.”
Gibson Env: Real-World Perception for Embodied Agents (2018, background): “On the how front, the problem has been the focus of a broad set of topics for decades, ranging from classical control [64, 12, 50] to more recently sensorimotor control [33, 54, 55, 5], reinforcement learning [6, 73, 74], acting by prediction [28], imitation learning [23], and other concepts [59, 10”
Reasoning About Physical Interactions with Object-Oriented Prediction and Planning (2018, methodology): “Comparing with representative video prediction (Lee et al., 2018; Babaeizadeh et al., 2018) and physical prediction (Ehrhardt et al., 2017; Mottaghi et al., 2016; Li et al., 2017; Lerer et al., 2016) methods, our approach achieves substantially better results at tasks that require building…”
Who Let the Dogs Out? Modeling Dog Behavior from Visual Data (2018, background): “[22] predict the future trajectories of objects according to Newtonian physics.”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, methodology): “We have therefore designed and used our benchmark to assess a wide range of approaches in physics learning and prediction, sampling representatives from several model types: vision models that make pixel-level future predictions via fully convolutional architectures, [23, 1, 36, 21, 35, 70, 40, 41, ”