Learning Physical Intuition of Block Towers by Example

By Adam Lerer, Sam Gross, Rob Fergus. 2016

320 citations recorded by the source corpus.

Wooden blocks are a common toy for infants, allowing them to develop motor skills and gain intuition about the physical behavior of the world. In this paper, we explore the ability of deep feedforward models to learn such intuitive physics. Using a 3D game engine, we create small towers of wooden blocks whose stability is randomized and render them collapsing (or remaining upright). This data allows us to train large convolutional network models which can accurately predict the outcome, as well as estimating the block trajectories. The models are also able to generalize in two important ways: (i) to new physical scenarios, e.g. towers with an additional block and (ii) to images of real wooden blocks, where it obtains a performance comparable to human subjects.

Subfields: Computer Vision and Pattern Recognition, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In AI for Physical Sciences

Citations in the corpus

Interaction Networks for Learning about Objects, Relations and Physics (2016, background): “Recent advances in convolutional neural 66 networks (CNNs) have led to efforts that learn to predict coarse-grained physical dynamics from 67 images [19, 17, 18].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “…all model-based reinforcement learning work, as well as work that learns predictive models of physics but does not utilize them for control (e.g., Lerer et al. (2016); Battaglia et al. (2016)), is outside the scope of this tutorial, and we focus primarily on work that performs both model-fitting…”
Simulation-assisted machine learning (2018, positive): “Similar in spirit to SimKern, although differing in details, combining simulation and machine learning has been used in physics to predict object behaviour (Lerer et al., 2016; Wu et al., 2015).”
CLEVRER: CoLlision Events for Video REpresentation and Reasoning (2019, background): “…2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from images (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017; Lerer et al., 2016; Mottaghi et al., 2016; Fragkiadaki et al., 2016), or from a symbolic, abstract representation of the environment (Battaglia et…”
Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; ”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “( Vondrick et al., 2016a) train a conditional CNN to predict future feature vectors, and several papers (e.g., (Lerer et al., 2016; Mottaghi et al., 2016; Wu et al., 2017; van Steenkiste et al., 2018; Fragkiadaki et al., 2016; Watters et al., 2017)) predict future object states.”
Flexible Neural Representation for Physics Prediction (2018, background): “However, future predictors operating directly on 2D image representations often fail to generate sharp object boundaries and struggle with occlusions and remembering objects when they are no longer visually observable [1, 17, 16, 28, 29, 33, 34, 19].”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “Purely unsupervised approaches for state-to-state prediction (Agrawal et al., 2016; Lerer et al., 2016; Michalski et al., 2014; Sutskever et al., 2009) that use raw visual inputs as state-descriptions have yet to rival these capabilities.”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, methodology): “There have been several efforts [18, 19] which used CNN-based models to predict whether a stack of blocks would fall.”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, methodology): “, 2015) formulates the problem as inference over the parameters of a symbolic physics engine, while the bottom-up approach (Agrawal et al., 2016; Fragkiadaki et al., 2015; Lerer et al., 2016; Li et al., 2016; Mottaghi et al., 2015; 2016; Sutskever et al., 2009) learns to directly map physical observ”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, background): “[32, 23, 14] proposed using Newtonian physics in combination with neural networks to predict the dynamics of objects in the future.”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “Most existing physics and video prediction models use feedforward architectures [17, 15] or feedforward encodings of the image [20].”
Deep Learning: A Critical Appraisal (2018, background): “Recent work by Lerer et al (2016), Watters and colleagues (2017), Tenenbaum and colleagues(Wu, Lu, Kohli, Freeman, & Tenenbaum, 2017) and Davis and myself (Davis, Marcus, & Frazier-Logue, 2017) suggest some competing approaches to how to think about this, within the domain of everyday physical…”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, background): “UETorch [14], Project Malmo [12], and SceneNet [11] are other examples of non-photo-realistic simulated environments.”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, methodology): “Similar frameworks have been proposed by [39], [40], [44], [42], [41], but the”
DeepMind Lab (2016, background): “Any of these systems can be used to generate static datasets for computer vision as described e.g., in Mahendran et al. (2016); Richter et al. (2016).”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Several of these are also interactive making them suitable to learn control, including [5, 30, 35, 39, 72].”