EPOpt: Learning Robust Neural Network Policies Using Model Ensembles

By Aravind Rajeswaran, Sarvjeet Ghotra, Sergey Levine, Balaraman Ravindran. 2016

400 citations recorded by the source corpus.

Sample complexity and safety are major challenges when learning policies with reinforcement learning for real-world tasks, especially when the policies are represented using rich function approximators like deep neural networks. Model-based methods where the real-world target domain is approximated using a simulated source domain provide an avenue to tackle the above challenges by augmenting real data with simulated data. However, discrepancies between the simulated source domain and the target domain pose a challenge for simulated training. We introduce the EPOpt algorithm, which uses an ensemble of simulated source domains and a form of adversarial training to learn policies that are robust and generalize to a broad range of possible target domains, including unmodeled effects. Further, the probability distribution over source domains in the ensemble can be adapted using data from target domain and approximate Bayesian methods, to progressively make it a better approximation. Thus, learning on a model ensemble, along with source domain adaptation, provides the benefit of both robustness and learning/adaptation.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “[36] explore different training strategies for learning from an ensemble of models, including adversarial training and adapting the ensemble distribution using data from the target domain, but also do not demonstrate successful real-world transfer.”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “We analyze the problem of “model exploitation,” which a number of recent works have raised as a primary challenge in model-based reinforcement learning (Rajeswaran et al., 2017; Clavera et al., 2018; Kurutach et al., 2018).”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “The closest to our approach are the methods from [30, 31, 32, 33, 34] that propose to iteratively learn simulation parameters and train policies.”
Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “A robust controller can be learned by injecting noise to the simulation [19], adding perturbations to the simulated robot [37], leveraging multiple simulators [3], using domain randomization [48] and dynamics randomization [29, 40, 34].”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, positive): “By training policies to succeed on the entire ensemble, we can acquire policies that are explicitly trained for robustness, similar in spirit to previously proposed model ensemble methods [30], [39].”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “While the main goal of this work is to study how modelfree RL can be used to learn complex policies directly on real hardware, we also evaluated training in simulation and transfer, employing randomization to allow for transfer [22], [23].”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, background): “robust policies through adversarial training schemes [22], [23].”
Towards Generalization and Simplicity in Continuous Control (2017, background): “Designing tasks with model ensembles (Rajeswaran et al., 2016) or perturbation forces during training could potentially further improve the robustness.”
Graph networks as learnable physics engines for inference and control (2018, background): “, 2017), learning policies that are robust to hidden property variations (Rajeswaran et al., 2016), and learning exploration strategies in uncertain settings (Schmidhuber, 1991; Sun et al.”
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping (2018, background): “Several recent works have looked at adapting to physical discrepancies explicitly [38], [39], [40], and incorporating these ideas into grasping is an exciting avenue for future work.”
Asymmetric Actor Critic for Image-Based Robot Learning (2017, background): “Making policies robust for physics adaptation [26]–[28] is also receiving interest, but these methods haven’t been shown to be powerful enough to work on real robots.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, background): “Previous work has tried to alleviate model-bias by characterizing the uncertainty of the models and learning a robust policy [6, 7, 8, 9, 10], often using ensembles to represent the posterior.”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, methodology): “[27] also proposed training a control policy using an ensemble of dynamic models.”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Nominal dynamics models based on knowledge of physics [1], or through learning [2], complements a growing body of work on successful simulation to reality transfer and system identification [3, 4, 5, 6, 7].”
Learning Powerful Policies by Using Consistent Dynamics Model (2019, methodology): “Previous work has tried to address the modelbias issue of model-based methods, by having a way to characterize the uncertainty of the models, and by learning a more robust policy (Deisenroth & Rasmussen, 2011; Rajeswaran et al., 2016; Lim et al., 2013).”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “There have been more robotics works that do not use vision, but that apply domain randomization on physical properties of the simulator to aid transferability [40, 46, 1, 68, 43].”
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control (2017, methodology): “(respectively), as these implementations are used in several works directly for comparison [12, 10, 9, 13, 14] Performance Measures : We examine the general variance of the algorithms and address the importance of presenting all possible metrics across a large number of trials.”
Deep reinforcement learning: a survey (2020, methodology): “The ensemble policy optimization algorithm (Rajeswaran et al., 2017) can adapt the probability over source domains in the ensemble us-ing the data from the target domain and randomizing physical parameters of simulated source domains.”
Mutual Alignment Transfer Learning (2017, background): “[20], who train robust policies based on sampling from an ensemble of simulated source domains while focusing on simulations with reduced performance.”
IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks (2019, background): “Once trained with enough variation, the policy should be able to generalize to the real world [30–32, 22, 5].”