Model-Ensemble Trust-Region Policy Optimization

By Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel. 2018

487 citations recorded by the source corpus.

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Model-Based Reinforcement Learning for Atari (2019, methodology): “In future work, models that capture uncertainty via Bayesian parameter posteriors or ensembles (Kurutach et al., 2018; Chua et al., 2018) may improve robustness.”
Learning Latent Dynamics for Planning from Pixels (2018, background): “…to develop hybrid agents that accelerate policy learning by training on imagined experience (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Kurutach et al., 2018; Buckman et al., 2018; Ha and Schmidhuber, 2018), improving feature representations (Wayne et al., 2018; Igl et al., 2018), or…”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “However, while these prior methods typically require additional machinery to work well, such as ensembles in the online setting (Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism mechanisms in the offline setting (Yu et al.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, background): “Ensembles of bootstrapped models are even simpler still: given a base model, no additional (hyper-)parameters need be tuned, whilst still providing reasonable uncertainty estimates [Efron and Tibshirani, 1994, Osband, 2016, Kurutach et al., 2018].”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “While there is no natural way to give a policy access to the ground truth model, we allow the policies to train on future trajectories generated via the ground truth model, in similar fashion to algorithms that learn a model for this purpose [5, 18], in order to help facilitate fair comparisons to m”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “In other applications such as model-based reinforcement learning (Deisenroth & Rasmussen, 2011; Wang et al., 2019), ensembles of neural networks can be used to estimate model uncertainty, leading to better overall performance (Kurutach et al., 2018).”
MOPO: Model-based Offline Policy Optimization (2020, background): “Uncertainty quantiﬁcation, a key ingredient to our approach, is critical to good performance in model-based RL both theoretically [62, 73, 43] and empirically [11, 6, 49, 38, 7], in optimal control [61, 2, 33].”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “As a result, planning using a learned model without any safeguards against model inaccuracy can result in “model exploitation” [30, 31, 29, 28], yielding poor results [32].”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “Ensembles have shown to be effective in preventing a policy or planning procedure from exploiting such inaccuracies (Rajeswaran et al., 2017; Kurutach et al., 2018; Clavera et al., 2018; Chua et al., 2018).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “All environments that we test have a maximum horizon of 500, which is longer than most of the existing model-based RL work (Nagabandi et al., 2017; Kurutach et al., 2018).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, neutral): “We refer to [Kurutach et al., 2018, Clavera et al., 2018, Sun et al., 2018b, Chua et al., 2018, Sanchez-Gonzalez et al., 2018, Pascanu et al., 2017, Khansari-Zadeh and Billard, 2011, Luo et al., 2018] and the reference therein for recent work on model-based RL. Off-policy reinforcement learning…”
Reinforcement Learning with Augmented Data (2020, methodology): “[43] Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel.”
Benchmarking Model-Based Reinforcement Learning (2019, methodology): “Dyna-Style Algorithms: MB-MPO surpasses the performance of ME-TRPO in most of the environments and achieves the best performance in domains like HalfCheetah.”
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion (2018, methodology): “[20] propose model-ensemble trust-region policy optimization (ME-TRPO), which is motivated similarly to this work in that they also propose an algorithm which uses an ensemble of models to mitigate the deleterious effects of model bias.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, methodology): “Previous work has tried to alleviate model-bias by characterizing the uncertainty of the models and learning a robust policy [6, 7, 8, 9, 10], often using ensembles to represent the posterior.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “While this capability has been demonstrated in some of the recent works (Levine et al., 2016; Nagabandi et al., 2018; Chua et al., 2018b; Kurutach et al., 2018; Ha & Schmidhuber, 2018), learning an accurate global model that works for all state-action pairs can be exceedingly challenging,…”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, methodology): “ME-TRPO (Kurutach et al., 2018), a model-based algorithm, performs a much lower 100-300 policy deployments, although this is still relatively high for practical settings.”
Deep Dynamics Models for Learning Dexterous Manipulation (2019, methodology): “Our method combines components from multiple prior works, including uncertainty estimation [23, 22, 27], deep models and model-predictive control (MPC) [19], and stochastic optimization for planning [28].”
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning (2018, methodology): “For the dynamics ˆ f , we use a neural network network with 8 layers of 128 neurons each with a ﬁxed 10 − 3 learning rate trained to predict the difference in real-vector-valued states, similar to previous work (Kurutach et al., 2018).”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “The improvement in sample efficiency from using model-based methods matches prior findings (Deisenroth and Rasmussen, 2011; Nagabandi et al., 2017a; Kurutach et al., 2018); the most important evaluation, which we discuss in more detail next, is the ability for our method to adapt online to drastic d”