Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control

By Sanket Kamthe, Marc Peter Deisenroth. 2017

231 citations recorded by the source corpus.

Trial-and-error based reinforcement learning (RL) has seen rapid advancements in recent times, especially with the advent of deep neural networks. However, the majority of autonomous RL algorithms require a large number of interactions with the environment. A large number of interactions may be impractical in many real-world applications, such as robotics, and many practical systems have to obey limitations in the form of state space or control constraints. To reduce the number of system interactions while simultaneously handling constraints, we propose a model-based RL framework based on probabilistic Model Predictive Control (MPC). In particular, we propose to learn a probabilistic transition model using Gaussian Processes (GPs) to incorporate model uncertainty into long-term predictions, thereby, reducing the impact of model errors. We then use MPC to find a control sequence that minimises the expected long-term cost. We provide theoretical guarantees for first-order optimality in the GP-based transition models with deterministic approximate inference for long-term planning. We demonstrate that our approach does not only achieve state-of-the-art data efficiency, but also is a principled way for RL in constrained environments.

Subfields: Control and Systems Engineering, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data (2020, background): “Accurate transition models for macroscopic physical systems are critical components in control systems (Lenz et al., 2015; Kamthe and Deisenroth, 2017; Chua et al., 2018) and data-efﬁcient reinforcement learning algorithms (Nagabandi et al., 2018; Janner et al., 2019).”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “the algorithm proposed by [54] except that we do not update the dynamics model after each transition, but only at the end of each trial.”
Benchmarking Model-Based Reinforcement Learning (2019, methodology): “Also, for PILCO to train for 200,000 time-steps, we have to use a data-set to increase training efficiency.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, background): “However, like Bayesian models [6, 20, 21], their application is limited to low-dimensional domains.”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “…has been traditionally approached with methods that are well-suited for low-data regimes such as Gaussian Processes (Rasmussen, 2003) showing signiﬁcant data-efﬁciency gains over model-free approaches (Deisenroth et al., 2013; Kamthe & Deisenroth, 2017; Kocijan et al., 2004; Ko et al., 2007).”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, methodology): “We refer to this baseline as SGP-I which is a sparse variant of the approach in [17] that achieves state-of-theart in data efficiency.”
Green AI (2019, background): “Data efficiency has received significant attention over the years [34, 19].”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “Learning Nonparametric Unknown Dynamics with Machine Learning Models.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “control [17, 132, 139, 183, 204, 231, 258], adaptive control [89, 115, 199, 238], robust control [26, 73, 112, 282], lyapunov function [56, 57] for RL stability and safety.”
Cautious Model Predictive Control Using Gaussian Process Regression (2017, background): “The most prominent example for tracking tasks is a quadratic cost on states and inputs; further examples include a saturating cost [24], risk sensitive costs [25], or radial basis function networks [9].”
Learning-Based Model Predictive Control: Toward Safe Learning in Control (2020, methodology): “Additionally, Kamthe & Deisenroth (88) showed that learning-based is a highly competitive model-based reinforcement learning technique, particularly with regard to data efficiency.”