Learning to Perform Physics Experiments via Deep Reinforcement Learning

By Misha Denil, Pulkit Agrawal, Tejas D. Kulkarni, Tom Erez, Peter W. Battaglia, Nando de Freitas. 2016

329 citations recorded by the source corpus.

When encountering novel objects, humans are able to infer a wide range of physical properties such as mass, friction and deformability by interacting with them in a goal driven way. This process of active interaction is in the same spirit as a scientist performing experiments to discover hidden facts. Recent advances in artificial intelligence have yielded machines that can achieve superhuman performance in Go, Atari, natural language processing, and complex control problems; however, it is not clear that these systems can rival the scientific intuition of even a young child. In this work we introduce a basic set of tasks that require agents to estimate properties such as mass and cohesion of objects in an interactive simulated environment where they can manipulate the objects and observe the consequences. We found that state of art deep reinforcement learning methods can learn to perform the experiments necessary to discover such hidden properties. By systematically manipulating the problem difficulty and the cost incurred by the agent for performing experiments, we found that agents learn different strategies that balance the cost of gathering information against the cost of making mistakes in different situations.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In AI for Physical Sciences

Citations in the corpus

Reinforcement Learning in Healthcare: A Survey (2019, background): “A more comprehensive and in-depth review on these issues can be found in Reference [238], and more recently in References [126, 213].”
Deep Q-learning From Demonstrations (2017, background): “…200 million steps with that of other deep reinforcement learning approaches: DQN, Double DQN, Prioritized DQN, Dueling DQN, PopArt, DQN+CTS, and DQN+PixelCNN (Mnih et al. 2015; van Hasselt, Guez, and Silver 2016; Schaul et al. 2016; Wang et al. 2016; van Hasselt et al. 2016; Ostrovski et al. 2017).”
Exploration by Random Network Distillation (2018, background): “Special access to the underlying RAM state can also be used to improve exploration by using it to hand-craft exploration bonuses (Kulkarni et al., 2016; Tang et al., 2017; Stanton & Clune, 2018).”
DeepMind Control Suite (2018, background): “Establishing baselines for reinforcement learning problems and algorithms is no-toriously diﬃcult (Islam et al., 2017; Henderson et al., 2017).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “The availability of high-quality visual renderers and physics engines now enables us to take steps in this direction, with works that try to learn intuitive models of physics in visual environments [23].”
Meta-learning of Sequential Strategies (2019, methodology): “…Garnelo et al., 2018, Gordon et al., 2018, Zintgraf et al., 2018, Chen et al., 2018a], or even the entire learning procedure using a memory-based architecture such as a recurrent neural network [Santoro et al., 2016, Wang et al., 2016, Duan et al., 2016, Denil et al., 2016, Mishra et al., 2018].”
Mind Games: Game Engines as an Architecture for Intuitive Physics. (2017, methodology): “Other networks have been trained to predict the e ﬀ ects of forces from still images [60, 61], and as part of an unsupervised action-guiding predictor of pixel-motion [62, 63] and physical properties [64].”
Neuroscience-Inspired Artificial Intelligence. (2017, methodology): “which uses predictions generated from an internal model of the environment learned through experience (Daw et al., 2005; Dolan and Dayan, 2013; Tolman, 1948).”
Exploring applications of deep reinforcement learning for real-world autonomous driving systems (2019, background): “inforcement Learning pipeline for autonomous driving is proposed in (Sallab et al., 2017), where the inputs are the states of the environment and their aggregations over time, and the output is the driving actions.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “Intelligent agents continuously learn to reﬁne their actions over states from the environment to maximize the accumulative reward functions [41], [42].”
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, background): “Nevertheless, there still exists a number of problems in the application of reinforcement learning in navigation such as reducing training time, storing data over long time, separating from computation, adapting rapidly to new environments etc [1].”
Model Identification via Physics Engines for Improved Policy Search (2017, background): “An increasingly popular alternative addresses these challenges through end-to-end learning [27]–[38].”
Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining (2020, methodology): “It has recently been applied to various optimization problems in structured input spaces [30], notably in chemical design [64, 65, 16, 43, 45, 51].”
dm_control: Software and Tasks for Continuous Control (2020, background): “The OpenAI Gym (Brockman et al., 2016) includes a set of continuous control domains that have become a popular benchmark in continuous RL (Duan et al., 2016; Henderson et al., 2017).”
Learning A Physical Long-term Predictor (2017, background): “Other approaches such as (Agrawal et al., 2016) or (Denil et al., 2016) also attempted to learn intuitive physics of objects through manipulation.”