Efficient reinforcement learning for robots using informative simulated priors

By Mark Cutler, Jonathan P. How. 2015

87 citations recorded by the source corpus.

Autonomous learning through interaction with the physical world is a promising approach to designing controllers and decision-making policies for robots. Unfortunately, learning on robots is often difficult due to the large number of samples needed for many learning algorithms. Simulators are one way to decrease the samples needed from the robot by incorporating prior knowledge of the dynamics into the learning algorithm. In this paper we present a novel method for transferring data from a simulator to a robot, using simulated data as a prior for real-world learning. A Bayesian nonparametric prior is learned from a potentially black-box simulator. The mean of this function is used as a prior for the Probabilistic Inference for Learning Control (PILCO) algorithm. The simulated prior improves the convergence rate and performance of PILCO by directing the policy search in areas of the state-space that have not yet been observed by the robot. Simulated and hardware results show the benefits of using the prior knowledge in the learning framework.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “It is often faster to ﬁne-tune a controller learned in simulation than to learn from scratch in the real world [7], [18].”
Model-based reinforcement learning with parametrized physical models and optimism-driven exploration (2015, neutral): “Several prior methods have suggested incorporating knowledge about the dynamics as a prior on the dynamics model [11], [12], [23].”
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, methodology): “In robotics, simulation to real-world generalization has been addressed using hierarchies of multi-ﬁdelity simulators [9], priors imposed on Bayesian dynamics models [8].”
Asymmetric Actor Critic for Image-Based Robot Learning (2017, background): “A recent approach to circumvent these challenges is to train on a simulated version of the robot and then transfer to the real robot [5]–[12].”
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors (2015, background): “A single attempt at the task is often insufficient to obtain such an accurate model, and while these challenges can be mitigated by incorporating domain knowledge about the system [3], [4] or demonstrations [5], they make the development of a generalpurpose one-shot learning method difficult.”
Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control (2017, methodology): “This approximation has been shown to work well in practice in RL contexts [9, 8, 7, 2, 3, 29, 30] and can be computed in closed form with the Gaussian kernel (3).”
Synthetic Data for Deep Learning (2019, background): “[129], the results of training on a simulator serve as a prior for subsequent learning in the real world.”
Fast Model Identification via Physics Engines for Data-Efficient Policy Search (2017, methodology): “Utilizing simulation to learn a prior before learning on the robot is a common way to reduce real-world data needs for policy search and provides data-efﬁciency over PILCO [Cutler and How, 2015].”
Probabilistically safe policy transfer (2017, background): “A number of recent papers have explored how to transfer policies from one domain to another [5], [6], [7], [11], [20], [24], [28], [29].”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, background): “…recent papers have explored the idea of incorporating physics priors into learning dynamics models of environments (Nguyen-Tuong and Peters, 2010; Cutler et al., 2014; Cutler and How, 2015; Scholz et al., 2014; Kansky et al., 2017; Battaglia et al., 2016; Mrowca et al., 2018; Xie et al., 2016).”
Doubly Stochastic Variational Inference for Deep Gaussian Processes (2017, background): “Examples include periodicities in climate modelling (Rasmussen and Williams, 2006), change-points in time series data (Garnett et al., 2009) and simulator priors for robotics (Cutler and How, 2015).”
PAC Reinforcement Learning With an Imperfect Model (2018, background): “A promising approach to addressing this issue is to train an RL agent in a simulator and transfer the solution to the real environment, which is particularly relevant but not limited to robotics domains (Koos, Mouret, and Doncieux 2010; Cutler and How 2015; Hanna and Stone 2017).”
Deep learning in bioinformatics (2016, background): “Currently, its applications are mainly focused on game playing [4] and robotics [225].”
Grounded Action Transformation for Robot Learning in Simulation (2017, background): “An alternative to learning directly on the robot is learning in simulation (Cutler and How 2015; Koos, Mouret, and Doncieux 2010).”