Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

By Kurtland Chua, Roberto Calandra, Rowan Thomas McAllister, Sergey Levine. 2018

1,543 citations recorded by the source corpus.

Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “For example, Chua et al. (2018); Doerr et al. (2018); Hafner et al. (2018); Buesing et al. (2018); Feinberg et al. (2018); Buckman et al. (2018) train models to predict a reward and a distribution over next states, minimizing the negative log-probability of the true next state.”
Model-Based Reinforcement Learning for Atari (2019, methodology): “In future work, models that capture uncertainty via Bayesian parameter posteriors or ensembles may further improve robustness (Kurutach et al., 2018; Chua et al., 2018).”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “Related Work Previous work in model-based reinforcement learning has focused on planning in low-dimensional state spaces (Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018), combining the benefits of model-based and model-free approaches (Kalweit & Boedecker, 2017; Nagab”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “(with uniform discretization) to that of the probabilistic feedforward model ensemble (Chua et al., 2018) over the course of a planning horizon in the humanoid environment, corresponding to the trajectories visualized in Figure 2.”
E(n) Equivariant Graph Neural Networks (2021, background): “Modelling these complex dynamics is crucial in a variety of applications such as control systems (Chua et al., 2018), model based dynamics in reinforcement learning (Nagabandi et al.”
A General Framework for Uncertainty Estimation in Deep Learning (2019, background): “Also for the task of robot manipulation, uncertainty estimation was shown to play a fundamental role to increase the learning efficiency and guarantee the manipulator safety [21], [22].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “, 2012) with various trajectory optimization methods (Nagabandi et al., 2018; Chua et al., 2018).”
Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation (2020, methodology): “Probabilistic Ensembles with Trajectory Sampling algorithm (PETS) [18] combine uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “, 2018), or combined with trajectory optimization (Chua et al., 2018) or valuebased methods (Buckman et al.”
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “Accurate transition models for macroscopic physical systems are critical components in control systems (Lenz et al., 2015; Kamthe and Deisenroth, 2017; Chua et al., 2018) and data-efﬁcient reinforcement learning algorithms (Nagabandi et al., 2018; Janner et al., 2019).”
Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “Past model-based planning procedures [6, 31] run a fixed number of iterations of MPC per timestep before executing an action in the environment.”
MOPO: Model-based Offline Policy Optimization (2020, background): “Uncertainty quantiﬁcation, a key ingredient to our approach, is critical to good performance in model-based RL both theoretically [62, 73, 43] and empirically [11, 6, 49, 38, 7], in optimal control [61, 2, 33].”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “These advantages have enabled MBRL to be highly sample efﬁcient in the online RL setting, often substantially outperforming model-free counterparts [28, 29, 30, 31].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Ensembles have shown to be effective in preventing a policy or planning procedure from exploiting such inaccuracies (Rajeswaran et al., 2017; Kurutach et al., 2018; Clavera et al., 2018; Chua et al., 2018).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, background): “The work of Chua et al. (2018) makes further improvement by using a probabilistic model ensemble.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “We refer to (Kurutach et al., 2018; Clavera et al., 2018; Sun et al., 2018b; Chua et al., 2018; Sanchez-Gonzalez et al., 2018; Pascanu et al., 2017; Khansari-Zadeh & Billard, 2011; Luo et al., 2018) and the reference therein for recent work on model-based RL.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “Our models are deep neural networks trained to maximize the log likelihood of the next state and reward given the current state and action, similar to models from successful model-based RL algorithms (Chua et al., 2018; Janner et al., 2019).”
Reinforcement Learning with Augmented Data (2020, methodology): “[41] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine.”
Benchmarking Model-Based Reinforcement Learning (2019, methodology): “Shooting Algorithms: RS is very effective on simple tasks such as InvertedPendulum, CartPole and Acrobot, but as task difficulty increases RS gradually gets surpassed by PETS-RS and PETS-CEM, which indicates that modelling uncertainty aware dynamics is crucial for the performance of shooting algorit”