Visual Stability Prediction and Its Application to Manipulation

By Wenbin Li, Ales Leonardis, Mario Fritz. 2016

38 citations recorded by the source corpus.

Understanding physical phenomena is a key competence that enables humans and animals to act and interact under uncertain perception in previously unseen environments containing novel objects and their configurations. Developmental psychology has shown that such skills are acquired by infants from observations at a very early stage. In this paper, we contrast a more traditional approach of taking a model-based route with explicit 3D representations and physical simulation by an {\em end-to-end} approach that directly predicts stability from appearance. We ask the question if and to what extent and quality such a skill can directly be acquired in a data-driven way---bypassing the need for an explicit simulation at run-time. We present a learning-based approach based on simulated data that predicts stability of towers comprised of wooden blocks under different conditions and quantities related to the potential fall of the towers. We first evaluate the approach on synthetic data and compared the results to human judgments on the same stimuli. Further, we extend this approach to reason about future states of such towers that in turn enables successful stacking.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Structured agents for physical construction (2019, background): “Several recent studies have explored learning construction, including learning to stack blocks by placing them at predicted stable points (Li et al., 2017), learning to attach blocks together to stabilize an unstable stack (Hamrick et al.”
Long-Term Image Boundary Prediction (2016, background): “(Fragkiadaki et al. 2016) predict future states of balls moving on a billiard table and (Lerer, Gross, and Fergus 2016; Li, Leonardis, and Fritz 2017) predict the stability of towers made out of blocks.”
Learning A Physical Long-term Predictor (2017, unknown): “In a different framework (Lerer et al., 2016) and (Li et al., 2016) also used the power of deep learning to extract an abstract representation of the concept of stability of block towers purely from images.”
IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning (2018, background): “Recent studies have investigated models for predicting the stability and forward modeling the dynamics of towers of blocks ([8, 35, 74, 39, 46, 40]).”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “Recent papers use neural networks over visual inputs to predict future pixels (Finn et al., 2016; Lerer et al., 2016; Mirza et al., 2016; Du and Narasimhan, 2019) or make qualitative predictions (Groth et al., 2018; Li et al., 2016, 2017; Janner et al., 2019; Wu et al., 2015; Mao et al., 2019).”
IntPhys: A Benchmark for Visual Intuitive Physics Reasoning (2019, background): “Some studies have investigated models for predicting the stability and forward modeling the dynamics of towers of blocks ([6, 25, 44, 26, 30, 27]).”
Long-Term Image Boundary Extrapolation (2016, background): “Developing an intuitive understanding of physics from raw visual input has been recently explored in [5, 12, 13].”
PHYRE: A New Benchmark for Physical Reasoning (2019, methodology): “Early computational instantiations of this framework used probabilistic models over physical simulations [3, 61], while more recent methods use feedforward neural networks trained to make pixelwise [11, 24, 34, 60] or qualitative [12, 26, 27, 34, 61] predictions about the future, sometimes in conjuc”