ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking

By Oliver Groth, Fabian B. Fuchs, Ingmar Posner, Andrea Vedaldi. 2018

119 citations recorded by the source corpus.

Physical intuition is pivotal for intelligent agents to perform complex tasks. In this paper we investigate the passive acquisition of an intuitive understanding of physical principles as well as the active utilisation of this intuition in the context of generalised object stacking. To this end, we provide ShapeStacks (Source code & data are available at http://shapestacks.robots.ox.ac.uk): a simulation-based dataset featuring 20,000 stack configurations composed of a variety of elementary geometric primitives richly annotated regarding semantics and structural stability. We train visual classifiers for binary stability prediction on the ShapeStacks data and scrutinise their learned physical intuition. Due to the richness of the training data our approach also generalises favourably to real-world scenarios achieving state-of-the-art stability prediction on a publicly available benchmark of block towers. We then leverage the physical intuition learned by our model to actively construct stable stacks and observe the emergence of an intuitive notion of stackability - an inherent object affordance - induced by the active stacking task. Our approach performs well exceeding the stack height observed during training and even manages to counterbalance initially unstable structures.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; ”
GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations (2019, methodology): “We None of models are able to reach the stability prediction accuracies in Groth et al. [10].”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, background): “[15], [16] focus on block stacking by classifying simulated stacks as stable or likely to fall.”
Explainable Deep Learning: A Field Guide for the Uninitiated (2020, methodology): “Specifically, in this study the experiments are conducted on the ShapeStacks dataset (Groth et al., 2018), which introduces a vision-based stability prediction task for block towers.”
PHYRE: A New Benchmark for Physical Reasoning (2019, methodology): “Early computational instantiations of this framework used probabilistic models over physical simulations [3, 61], while more recent methods use feedforward neural networks trained to make pixelwise [11, 24, 34, 60] or qualitative [12, 26, 27, 34, 61] predictions about the future, sometimes in conjuc”
RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces (2020, methodology): “To this, we add two popular synthetic datasets CLEVR [16] (cluttered tabletops) and ShapeStacks [12] (block stacking).”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, background): “Recent papers use neural networks over visual inputs to predict future pixels (Finn et al., 2016; Lerer et al., 2016; Mirza et al., 2016; Du and Narasimhan, 2019) or make qualitative predictions (Groth et al., 2018; Li et al., 2016, 2017; Janner et al., 2019; Wu et al., 2015; Mao et al., 2019).”
The CoSTAR Block Stacking Dataset: Learning with Workspace Constraints (2018, background): “This pairs well with ShapeStacks[6] a synthetic dataset for understanding how stacks of simple objects stand or fall.”
Explainable Machine Learning for Scientific Insights and Discoveries (2019, methodology): “Moreover, [67] provides a first step towards a physics-aware model by using their trained stability predictor and heatmap analysis to provide stackability scores for unseen object sets, for the estimation of an optimal placement of blocks, and to counterbalance instabilities by placing additional ob”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “ShapeStacks [41] presents a 3D simulation-based dataset with over 20,000 stack conﬁgurations of objects with elementary geometric shapes.”
Neural Production Systems (2021, methodology): “We use the shapes stack dataset introduced by Groth et al. (2018).”