"What Happens If..." Learning to Predict the Effect of Forces in Images

By Roozbeh Mottaghi, Mohammad Rastegari, Abhinav Kumar Gupta, Ali Farhadi. 2016

129 citations recorded by the source corpus.

What happens if one pushes a cup sitting on a table toward the edge of the table? How about pushing a desk against a wall? In this paper, we study the problem of understanding the movements of objects as a result of applying external forces to them. For a given force vector applied to a specific location in an image, our goal is to predict long-term sequential movements caused by that force. Doing so entails reasoning about scene geometry, objects, their attributes, and the physical rules that govern the movements of objects. We design a deep neural network model that learns long-term sequential dependencies of object movements while taking into account the geometry and appearance of the scene by combining Convolutional and Recurrent Neural Networks. Training our model requires a large-scale dataset of object movements caused by external forces. To build a dataset of forces in scenes, we reconstructed all images in SUN RGB-D dataset in a physics simulator to estimate the physical movements of objects caused by external forces applied to them. Our Forces in Scenes (ForScene) dataset contains 65,000 object movements in 3D which represent a variety of external forces applied to different types of objects. Our experimental evaluations show that the challenging task of predicting long-term movements of objects as their reaction to external forces is possible from a single image. The code and dataset are available at: http://allenai.org/plato/forces.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Mottaghi et al. (2016) predict the effects of forces on objects in images.”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Related to physical commonsense, research in visual commonsense has studied intuitive physics (Wu et al. 2017), cause-effect relationships (Mottaghi et al. 2016), and what can be reasonably inferred beyond a single image (Zellers et al. 2019a).”
From Recognition to Cognition: Visual Commonsense Reasoning (2018, neutral): “Commonsense Reasoning Our task unifies work involving reasoning about commonsense phenomena, such as physics [54, 84], social interactions [2, 77, 12, 27], procedure understanding [91, 3] and predicting what might happen next in a video [74, 17, 92, 78, 18, 64, 85].”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “, 2018), either directly from images (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017; Lerer et al., 2016; Mottaghi et al., 2016; Fragkiadaki et al., 2016), or from a symbolic, abstract representation of the environment (Battaglia et al.”
Flexible Neural Representation for Physics Prediction (2018, background): “However, future predictors operating directly on 2D image representations often fail to generate sharp object boundaries and struggle with occlusions and remembering objects when they are no longer visually observable [1, 17, 16, 28, 29, 33, 34, 19].”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, background): “[19, 20] predicted coarse, image-space motion trajectories of objects in real images.”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “…over the parameters of a symbolic physics engine, while the bottom-up approach (Agrawal et al., 2016; Fragkiadaki et al., 2015b; Lerer et al., 2016; Li et al., 2016; Mottaghi et al., 2015; 2016; Sutskever et al., 2009) learns to directly map observations to motion prediction or physical judgments.”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “Learning physics: Several works have explicitly addressed prediction of physical interactions, including predicting ball motion [5], block falling [2], the effects of forces [19, 18], future human interactions [9], and future car trajectories [26].”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, background): “main advantages of our framework are as follows: (1) The physics engine and the deep learning framework directly communicate (in contrast to separating the physics engine from the controller as in [38]).”
IQA: Visual Question Answering in Interactive Environments (2017, background): “isual Learning by Simulation: There has been an increased use of simulated environments and game platforms to train computer vision systems to perform tasks such as learning the dynamics of the world [47,48,70], semantic segmentation [18], pedestrian detection [43], pose estimation [53] and urban dr”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, neutral): “This is the approach taken by Mottaghi et al. (2016), who generate an “interactable” data set of scenes, which they use to generate a static data set of image and force pairs, along with the ground truth trajectory of a target object in response to the application of the indicated force.”
SE3-nets: Learning rigid body motion using deep neural networks (2016, methodology): “Physics prediction: Our model is related to recent work in deep learning for physics based prediction, such as predicting the stability of a tower of blocks from images [19, 20], predicting the motion of a billiards ball under forces [10] and predicting the dynamics of objects in images [23, 24].”
Visual Semantic Planning Using Deep Successor Representations (2017, background): “Some early successes have been shown in physical understanding [9, 26, 28, 36] by interacting with an environment.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Some other related works [59, 1, 63, 44] extend physical models to predict the effect of forces and infer the geometric attributes and topological relations of cuboids.”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, methodology): “We have therefore designed and used our benchmark to assess a wide range of approaches in physics learning and prediction, sampling representatives from several model types: vision models that make pixel-level future predictions via fully convolutional architectures, [23, 1, 36, 21, 35, 70, 40, 41, ”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “[24, 25, 46, 47] predict the dynamics of objects and their future trajectory.”