Physics 101: Learning Physical Object Properties from Unlabeled Videos

By Jiajun Wu, Joseph J. Lim, Hongyi Zhang, Josh Tenenbaum, William T. Freeman. 2016

162 citations recorded by the source corpus.

We study the problem of learning physical properties of objects from unlabeled videos. Humans can learn basic physical laws when they are very young, which suggests that such tasks may be important goals for computational vision systems. We consider various scenarios: objects sliding down an inclined surface and colliding; objects attached to a spring; objects falling onto various surfaces, etc. Many physical properties like mass, density, and coefficient of restitution influence the outcome of these scenarios, and our goal is to recover them automatically. We have collected 17,408 video clips containing 101 objects of various materials and appearances (shapes, colors, and sizes). Together, they form a dataset, named Physics 101, for studying object-centered physical properties. We propose an unsupervised representation learning model, which explicitly encodes basic physical laws into the structure and use them, with automatically discovered observations from videos, as supervision. Experiments demonstrate that our model can learn physical properties of objects from video. We also illustrate how its generative nature enables solving other tasks such as outcome prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016;…”
Flexible Neural Representation for Physics Prediction (2018, methodology): “More similar to our approach are inverse graphics methods that extract a lower dimensional physical representation from images that is used to predict physics [25, 26, 51, 50, 52, 53, 7, 49].”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, background): “Several efforts [4, 6, 26, 27] have fit the parameters of Newtonian mechanics equations to systems depicted in images and videos, though the dynamic equations themselves were not learned.”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “…strength, but automatically inferring from visual data what physical laws and object properties are present requires more work in inverse graphics (Chen et al., 2016; Kulkarni et al., 2014; 2015a;b; Whitney et al., 2016) and physics-based visual understanding (Brand, 1997; Wu et al., 2015; 2016).”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, background): “Physics 101 (Wu et al., 2016) is an extension of Galileo that more fully embraces deep learning.”
From Lifestyle Vlogs to Everyday Interactions (2017, background): “This has obvious applications in tasks like future prediction [41], intuitive physics [44], imitation learning from videos [45], and grasp analysis [18].”
Learning A Physical Long-term Predictor (2017, methodology): “Works of (Wu et al., 2015) and its extension (Wu et al., 2016) propose methods to learn physical properties of scenes and objects.”
Generating Liquid Simulations with Deformation-aware Neural Networks (2017, background): “Others have targeted this goal in specific settings such as robotic interactions [14], sliding and colliding objects [43, 42], billiard [15], or trajectories in height fields [12].”
Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense (2020, background): “[230] Jiajun Wu, Joseph J Lim, Hongyi Zhang, Joshua B Tenenbaum, William T Freeman, Physics 101: Learning physical object properties from unlabeled videos, in: British Machine Vision Conference (BMVC), 2016.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, methodology): “In addition to object recognition, passive perception is often used to estimate the position, shape, and material properties [244, 30, 419, 392, 120, 335, 48, 169, 49, 422, 426].”
Interpretable Intuitive Physics Model (2018, background): “For example, Physics101 dataset [24] collects sequences of collisions for this task.”