To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction

By Wenbin Li, Seyedmajid Azimi, Ales Leonardis, Mario Fritz. 2016

71 citations recorded by the source corpus.

Understanding physical phenomena is a key competence that enables humans and animals to act and interact under uncertain perception in previously unseen environments containing novel object and their configurations. Developmental psychology has shown that such skills are acquired by infants from observations at a very early stage. In this paper, we contrast a more traditional approach of taking a model-based route with explicit 3D representations and physical simulation by an end-to-end approach that directly predicts stability and related quantities from appearance. We ask the question if and to what extent and quality such a skill can directly be acquired in a data-driven way bypassing the need for an explicit simulation. We present a learning-based approach based on simulated data that predicts stability of towers comprised of wooden blocks under different conditions and quantities related to the potential fall of the towers. The evaluation is carried out on synthetic data and compared to human judgments on the same stimuli.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition, Developmental and Educational Psychology, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

In Developmental and Educational Psychology

In Robotics & Control

Citations in the corpus

Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; ”
Flexible Neural Representation for Physics Prediction (2018, background): “However, future predictors operating directly on 2D image representations often fail to generate sharp object boundaries and struggle with occlusions and remembering objects when they are no longer visually observable [1, 17, 16, 28, 29, 33, 34, 19].”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, methodology): “There have been several eﬀorts [17, 18] which used CNN-based models to predict whether a stack of blocks would fall.”
Learning Physical Intuition of Block Towers by Example (2016, neutral): “These models are purely bottom-up in nature, in contrast to existing approaches which rely on complex top-down graphics engines.”
SE3-nets: Learning rigid body motion using deep neural networks (2016, methodology): “These methods mostly predict low-dimensional outputs like ball velocity [10] or the probability of falling [20].”
Visual Semantic Planning Using Deep Successor Representations (2017, background): “Some early successes have been shown in physical understanding [9, 26, 28, 36] by interacting with an environment.”
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning (2018, methodology): “Recent studies have investigated models for predicting the stability and forward modeling the dynamics of towers of blocks ([8, 35, 74, 39, 46, 40]).”
Hierarchical Relational Inference (2020, background): “Several such approaches were also proposed in the context of physical reasoning (Finn, Goodfellow, and Levine 2016; Lerer, Gross, and Fergus 2016; Li et al. 2016).”
A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understanding (2016, unknown): “…in both cognitive science and computational vision (Battaglia, Hamrick, & Tenenbaum, 2013; Gupta, Efros, & Hebert, 2010; Mot-taghi, Bagherinezhad, Rastegari, & Farhadi, 2015; Fragki-adaki, Agrawal, Levine, & Malik, 2015; Zheng, Zhao, Yu, Ikeuchi, & Zhu, 2015; Li, Azimi, Leonardis, & Fritz, 2016).”
Relational inductive bias for physical construction in humans and machines (2018, methodology): “Our approach contrasts with standard deep learning approaches to physical reasoning, which are often computed holistically over a fixed representation and do not explicitly have a notion of objects or relations (e.g. Lerer et al., 2016; W. Li et al., 2016).”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “These models can range from physics-based prediction models using pixel-level convolutional networks [31, 32, 33, 34, 29, 35] and scene-based learning [14, 36, 37, 38] to learning of physical representations of the intuitive physics [6, 2, 39] behind the interactions to make accurate prediction of…”
Interpretable Intuitive Physics Model (2018, background): “The underlying goal of most of these systems is to use physics to predict what is going to happen next [8,6,13,14,7,26,24].”
Interaction Networks for Learning about Objects, Relations and Physics (2016, background): “Recent advances in convolutional neural 66 networks (CNNs) have led to efforts that learn to predict coarse-grained physical dynamics from 67 images [19, 17, 18].”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “Recent papers use neural networks over visual inputs to predict future pixels (Finn et al., 2016; Lerer et al., 2016; Mirza et al., 2016; Du and Narasimhan, 2019) or make qualitative predictions (Groth et al., 2018; Li et al., 2016, 2017; Janner et al., 2019; Wu et al., 2015; Mao et al., 2019).”
Explainable Machine Learning for Scientific Insights and Discoveries (2019, background): “In a similar way, but with more complex scenes or differently shaped objects, [40] predicts the physical stability of stacked objects using various popular convolutional neural network (CNN) architectures.”
ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking (2018, methodology): “[7,20] is related to ours in that a stability predictor is trained for Kappla blocks in simulation which is then applied to guide stacking with a robotic arm.”
PHYRE: A New Benchmark for Physical Reasoning (2019, methodology): “Early computational instantiations of this framework used probabilistic models over physical simulations [3, 61], while more recent methods use feedforward neural networks trained to make pixelwise [11, 24, 34, 60] or qualitative [12, 26, 27, 34, 61] predictions about the future, sometimes in conjuc”
IntPhys: A Benchmark for Visual Intuitive Physics Reasoning (2019, methodology): “[6] proposes a model based on an intuitive physics engine, [25] and [26] follow a supervised approach using Convolutional Neural Networks (CNNs), [44] makes a comparison between simulation-based models and CNN-based models, [30] improves the predictions of a CNN model by providing it with a…”