Improving PILCO with Bayesian Neural Network Dynamics Models

By Rowan Thomas McAllister, Carl E. Rasmussen. 2016

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, methodology): “Related Work Previous work in model-based reinforcement learning has focused on planning in low-dimensional state spaces (Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018), combining the benefits of model-based and model-free approaches (Kalweit & Boedecker, 2017; Nagab”
Recurrent World Models Facilitate Policy Evolution (2018, background): “Other recent works [12, 17] use Bayesian neural networks instead of GPs to learn a dynamics model.”
Weight Agnostic Neural Networks (2019, background): “Our environment is closely based on the one described in [27, 120].”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “This is used to combat the well-known problems with compounding model errors, and is present in policy search methods (Deisenroth & Ras-mussen, 2011; Gal et al., 2016; Higuera et al., 2018), combined with trajectory optimization (Chua et al., 2018) or value-based methods (Buckman et al., 2018).”
World Models (2018, background): “Other recent works (Gal et al., 2016; Depeweg et al., 2016) use Bayesian neural networks instead of GPs to learn a dynamics model.”
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (2018, methodology): “can represent multimodal distributions, forcing a unimodal distribution via moment matching (MM) can (in some cases) benefit MBRL data efficiency [12].”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, positive): “[31] extended PILCO [6] using Bayesian neural”
Concrete Dropout (2017, methodology): “Lastly, we demonstrate our dropout variant in a model-based RL task extending on [10], showing that the agent correctly reduces its uncertainty dynamically as the amount of data increases (§4.”
Uncertainty-Aware Reinforcement Learning for Collision Avoidance (2017, methodology): “Recent work has proposed to use a Bayesian formulation of neural networks based on dropout [8], as well as to use the bootstrap for exploration [22], but not, to the best of our knowledge, for uncertainty estimation for the purpose of safety.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “Our approach builds upon the wealth of prior work on model-based online RL methods that models the dynamics by Gaussian processes [11], local linear models [41, 37], neural network function approximators [14, 20, 13], and neural video prediction models [15, 31].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Neural network predictive models (Draeger et al., 1995; Gal et al., 2016; Depeweg et al., 2016; Nagabandi et al., 2018), are appealing because they allow for algorithms that combine the sample efﬁciency of a model-based approach with the asymptotic performance of high-capacity function…”
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “Such an explanation was previously provided by Gal et al. (2016).”
Model-Ensemble Trust-Region Policy Optimization (2018, neutral): “…studies have shown promising results on relatively simple domains (Nguyen & Widrow, 1990; Schmidhuber & Huber, 1991; Jordan & Rumelhart, 1992; Gal et al., 2016), so far their applications on more challenging domains have either required a combination with model-free techniques (Oh et al.,…”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “This narrative has proved popular despite the fact that dropout distribution can be a poor approximation to most reasonable Bayesian posteriors [22, 46]: Lemma 1 (Dropout distribution does not concentrate with observed data).”
Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks (2016, methodology): “, 2016), which has recently been used to train Bayesian neural networks in reinforcement learning problems (Blundell et al., 2015; Houthooft et al., 2016; Gal et al., 2016).”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “More recently, due to the challenges of applying these methods to high-dimensional state spaces, MBRL approaches employs Bayesian deep neural networks (Nagabandi et al., 2018; Chua et al., 2018b; Gal et al., 2016; Fu et al., 2016; Lenz et al., 2015) to learn dynamics models.”
Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control (2019, background): “While the merits of this decomposition have been demonstrated in low-dimensional environments (Deisenroth & Rasmussen, 2011; Gal et al., 2016), scaling these methods to high-dimensional environments remains an open challenge.”
Efficient exploration with Double Uncertain Value Networks (2017, methodology): “For this paper we consider the Bayesian dropout (Gal et al., 2016), as it has a very simple practical implementation (see Sec.”
Mastering Atari with Discrete World Models (2020, background): “World models Several model-based agents focus on proprioceptive inputs (Watter et al., 2015; Gal et al., 2016; Higuera et al., 2018; Henaff et al., 2018; Chua et al., 2018; Wang et al., 2019; Wang and Ba, 2019), model images without using them for planning (Oh et al., 2015; Krishnan et al., 2015;…”