Reinforcement learning with multi-fidelity simulators

By Mark Cutler, Thomas J. Walsh, Jonathan P. How. 2014

93 citations recorded by the source corpus.

We present a framework for reinforcement learning (RL) in a scenario where multiple simulators are available with decreasing amounts of fidelity to the real-world learning scenario. Our framework is designed to limit the number of samples used in each successively higher-fidelity/cost simulator by allowing the agent to choose to run trajectories at the lowest level that will still provide it with information. The approach transfers state-action Q-values from lower-fidelity models as heuristics for the “Knows What It Knows” family of RL algorithms, which is applicable over a wide range of possible dynamics and reward representations. Theoretical proofs of the framework's sample complexity are given and empirical results are demonstrated on a remote controlled car with multiple simulators. The approach allows RL algorithms to find near-optimal policies for the real world with fewer expensive real-world samples than previous transfer approaches or learning without simulators.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Iterative learning control has been applied to a variety of robotic control problems, from model car control (e.g., [1] and [8]) to surgical robotics (e.g., [48]).”
Model-based reinforcement learning with parametrized physical models and optimism-driven exploration (2015, neutral): “Several prior methods have suggested incorporating knowledge about the dynamics as a prior on the dynamics model [11], [12], [23].”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Multi-fidelity reinforcement learning (MFRL) framework [99] showed to transfer heuristics to guide exploration in high”
(CAD)$^2$RL: Real Single-Image Flight without a Single Real Image (2016, methodology): “In robotics, simulation to real-world generalization has been addressed using hierarchies of multi-ﬁdelity simulators [9], priors imposed on Bayesian dynamics models [8].”
Multi-fidelity Bayesian Optimisation with Continuous Approximations (2017, background): “As a result, multi-ﬁdelity optimisation meth-ods have recently gained attention (Cutler et al., 2014; Kan-dasamy et al., 2016a; Li et al., 2016).”
Synthetic Data for Deep Learning (2019, unknown): “Moreover, in [130, 131] Cutler et al.”
Reducing hardware experiments for model learning and policy optimization (2015, background): “Cutler et al. [17] trained a policy in multiple fidelity simulators with discretized actions.”
AutoRally: An Open Platform for Aggressive Autonomous Driving (2019, methodology): “A framework with multifidelity simulation and accompanying hardware platform for use in reinforcement-learning problems relating to autonomous driving was presented in [12].”
Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system (2018, background): “Transferring to real hardware can take many approaches as well, either through adaptation [32], [33], or incorporating uncertainty [34], [35].”
Task-Agnostic Dynamics Priors for Deep Reinforcement Learning (2019, background): “…recent papers have explored the idea of incorporating physics priors into learning dynamics models of environments (Nguyen-Tuong and Peters, 2010; Cutler et al., 2014; Cutler and How, 2015; Scholz et al., 2014; Kansky et al., 2017; Battaglia et al., 2016; Mrowca et al., 2018; Xie et al., 2016).”
Probabilistically safe policy transfer (2017, background): “A number of recent papers have explored how to transfer policies from one domain to another [5], [6], [7], [11], [20], [24], [28], [29].”
Grounded Action Transformation for Robot Learning in Simulation (2017, background): “Cutler et al. (2014) use lower ﬁdelity simulators to narrow the action search space for faster learning in higher ﬁdelity simulators or the real world.”
Learning legged swimming gaits from experience (2015, unknown): “This choice is inspired by authors such as Xu et al. [15], [16] who have recently shown that human guidance, in the forms of demonstrations or direct participation, are effective aids for learning practical robotic tasks.”
The Multi-fidelity Multi-armed Bandit (2016, methodology): “Multi-fidelity methods are used in the robotics community for reinforcement learning tasks by modeling each fidelity as a Markov decision process [6].”
Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations (2016, background): “Some literature have analysed multi-ﬁdelity methods in speciﬁc contexts such as hyper-parameter tuning, active learning and reinforcement learning [1, 5, 26, 33].”
Multi-Fidelity Black-Box Optimization with Hierarchical Partitions (2018, background): “This motivates studying the multi-fidelity setting where we have cheaper, but potentially biased approximations to the function (Cutler et al., 2014; Huang et al., 2006a; Kandasamy et al., 2016c; 2017).”
Efficient reinforcement learning for robots using informative simulated priors (2015, methodology): “In this paper we introduce a method for using samples from a simulated robot to decrease the number of real-world samples that are needed to learn a good policy (see Figure 1).”
Multi-fidelity Gaussian Process Bandit Optimisation (2016, methodology): “In optimal policy search in robotics and autonomous driving, vastly cheaper computer simulations are used to approximate the expensive real world performance of the system (Cutler et al., 2014; Urmson et al., 2008).”