Gaussian Processes for Data-Efficient Learning in Robotics and Control

By Marc Peter Deisenroth, Dieter Fox, Carl E. Rasmussen. 2015

745 citations recorded by the source corpus.

Autonomous learning has been a promising direction in control and robotics for more than a decade since data-driven learning allows to reduce the amount of engineering knowledge, which is otherwise required. However, autonomous reinforcement learning (RL) approaches typically require many interactions with the system to learn controllers, which is a practical limitation in real systems, such as robots, where many interactions can be impractical and time consuming. To address this problem, current learning approaches typically require task-specific knowledge in form of expert demonstrations, realistic simulators, pre-shaped policies, or specific knowledge about the underlying dynamics. In this paper, we follow a different approach and speed up learning by extracting more information from data. In particular, we learn a probabilistic, non-parametric Gaussian process transition model of the system. By explicitly incorporating model uncertainty into long-term planning and controller learning our approach reduces the effects of model errors, a key problem in model-based learning. Compared to state-of-the art RL our model-based policy search method achieves an unprecedented speed of learning. We demonstrate its applicability to autonomous learning in real robot and control tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

From Pixels to Torques: Policy Learning with Deep Dynamical Models (2015, methodology): “There are two solutions to this problem: Either we use a probabilistic dynamics model as suggested in (Schneider, 1997; Deisenroth et al., 2015) to explicitly account for model uncertainty and the implied natural exploration or we follow an explicit exploration strategy to ensure proper excitation o”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “successfully used in MBRL [8], experimental results [52] suggest that particle approaches can be competitive both computationally and in terms of accuracy, without making strong assumptions about the distribution used.”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have tried to address this by developing simulations from which large amounts of diverse data can be collected [1, 2], or by attempting to make robot learning algorithms more data efﬁcient [3, 4].”
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “We performed learning tasks from a recent PILCO paper (Deisenroth et al., 2015): cart-pole swing-up and balancing, and unicycle balancing.”
An Algorithmic Perspective on Imitation Learning (2018, methodology): “The marginal distribution p(xt+1|µt, Σt) in (3.75) can be approximated by a Gaussian distribution by following the results from [Deisenroth and Rasmussen, 2011, Deisenroth et al., 2013a].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, methodology): “…has been traditionally approached with methods that are well-suited for low-data regimes such as Gaussian Processes (Rasmussen, 2003) showing signiﬁcant data-efﬁciency gains over model-free approaches (Deisenroth et al., 2013; Kamthe & Deisenroth, 2017; Kocijan et al., 2004; Ko et al., 2007).”
Learning deep dynamical models from image pixels (2014, background): “In a setting where we make decisions based on (onestep or multiple-step ahead) predictions, such as optimal control or model-based reinforcement learning, a probabilistic model is often needed for robust decision making as we need to account for models errors [13, 3].”
Meta Reinforcement Learning with Latent Variable Gaussian Processes (2018, methodology): “We define yt = xt+1 − xt as the targets of the GP and take the mean function to be m(x̃t) = 0, which encodes that a priori the state does not change [9].”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, methodology): “Model-based RL methods have achieved significant efficiency benefits compared to model-free RL methods [9, 30, 10].”
Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control (2017, methodology): “The state space consists of 2 angles and 2 angular velocities [ θ 1 , θ 2 , ˙ θ 1 , ˙ θ 2 ] [8].”
Black-box data-efficient policy search for robotics (2017, methodology): “Both the model-based PGPE [30] and the PILCO [13] algorithm use gradient-based policy updates.”
Synthesizing Neural Network Controllers with Probabilistic Model-Based Reinforcement Learning (2018, methodology): “PILCO [1] uses Gaussian Process (GP) models to fit onestep dynamics and networks of radial basis functions (RBFs) as feedback policies.”
Optimism-driven exploration for nonlinear systems (2015, methodology): “We simulated the benchmark systems (pendulum, cartpole, double pendulum) using the same dynamics and parameters as the competing approach [23].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, methodology): “Though we could use any RL algorithm with our classifierbased reward, model-based RL has typically achieved greater data efficiency than model-free methods [18], [19], and the data efficiency afforded by the combination of model-based RL and representation learning further reduces the human supervis”
Planning and Execution using Inaccurate Models with Provable Guarantees (2020, background): “Existing planning and learning approaches try to learn a very good approximation of M from scratch through online executions [18, 7, 16, 11], or update the dynamics of model M̂ so that it approximates M well [1, 15, 26].”
Model-based contextual policy search for data-efficient generalization of robot skills (2017, methodology): “To improve the data-efficiency of model free methods, several model-based approaches have been proposed in the literature [9, 12, 34, 4].”
Uncertainty in Deep Learning (2016, background): “This means that PILCO underestimates state uncertainty at future time steps [Deisenroth et al., 2015], which can lead to diminished performance.”