Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning

By Jiajun Wu, Ilker Yildirim, Joseph J. Lim, Bill Freeman, Josh Tenenbaum. 2015

409 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In AI for Physical Sciences

Citations in the corpus

Simulation-assisted machine learning (2018, positive): “Similar in spirit to SimKern, although differing in details, combining simulation and machine learning has been used in physics to predict object behaviour (Lerer et al., 2016; Wu et al., 2015).”
Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016; ”
Flexible Neural Representation for Physics Prediction (2018, methodology): “More similar to our approach are inverse graphics methods that extract a lower dimensional physical representation from images that is used to predict physics [25, 26, 51, 50, 52, 53, 7, 49].”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “The former group performs inference over the parameters of a symbolic physics engine (Battaglia et al., 2013; Ullman et al., 2017; Wu et al., 2015), which restricts them to synthetic environments.”
Visual Interaction Networks: Learning a Physics Simulator from Video (2017, background): “Several efforts [4, 6, 26, 27] have fit the parameters of Newtonian mechanics equations to systems depicted in images and videos, though the dynamic equations themselves were not learned.”
Unsupervised Learning of 3D Structure from Images (2016, methodology): “In our experiments, context is either 1) nothing, 2) an object class label, or 3) one or more views of the scene from different cameras.”
The limits and potentials of deep learning for robotics (2018, background): “…grounded in real world experience, are well suited for closed-loop control, and can be learned and Recent approaches to end-to-end training of deep networks forgo the use of explicit physics models, learning predictive models and controls from raw experiences [9], [16], [18], [36], [129], [132].”
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning (2017, background): “While the focus in this work is to define a generative model for high dimensional videos of simple physical systems, several recent works have combined physical models of the world with deep learning to learn the dynamics of objects in more complex but low-dimensional environments [3, 4, 9, 34].”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, background): “…is to represent the world in terms of a ﬁxed set of physical parameters such as mass, friction coefﬁcient, normal forces etc and use a physics simulator for computing object dynamics (Kolev & Todorov, 2015; Mottaghi et al., 2015; Wu et al., 2015; Hamrick et al., 2011) from this representation.”
Attend, Infer, Repeat: Fast Scene Understanding with Generative Models (2016, background): “Hinton et al. (1995); Tu & Zhu (2002); Kulkarni et al. (2015a); Jampani et al. (2015); Wu et al. (2015) exploit data-driven predictions to empower the ‘vision as inverse graphics’ paradigm.”
Target-driven visual navigation in indoor scenes using deep reinforcement learning (2016, background): “Recently, physics engines have been used to learn the dynamics of real-world scenes from images [36], [37], [38].”
Learning Visual Predictive Models of Physics for Playing Billiards (2015, methodology): “Learning Physics from Images and Videos Works of Wu et al. (2015); Bhat et al. (2002); Brubaker et al. (2009); Mottaghi et al. (2015) propose methods for estimating the parameters of Newtonian equations from images and videos.”
Single Image 3D Interpreter Network (2016, methodology): “obust to imperfect detection results and can be jointly learned with keypoint estimators. Our work also connects to the traditional eld of vision as inverse graphics [14,21] and analysis by synthesis [58,20,5,52], as we use neural nets to decode latent 3D structure from images, and use a projection ”
IQA: Visual Question Answering in Interactive Environments (2017, background): “Visual Learning by Simulation: There has been an increased use of simulated environments and game platforms to train computer vision systems to perform tasks such as learning the dynamics of the world [47, 48, 70], semantic segmentation [18], pedestrian detection [43], pose estimation [53] and urban”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, background): “[39] learn a model for estimating physical properties of objects such as mass and friction based on a series of videos that show movement of objects on an inclined surface.”