Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks

By Stefan Depeweg, José Miguel Hernández-Lobato, Finale Doshi-Velez, Steffen Udluft. 2016

168 citations recorded by the source corpus.

We present an algorithm for model-based reinforcement learning that combines Bayesian neural networks (BNNs) with random roll-outs and stochastic optimization for policy learning. The BNNs are trained by minimizing $\alpha$-divergences, allowing us to capture complicated statistical patterns in the transition dynamics, e.g. multi-modality and heteroskedasticity, which are usually missed by other common modeling approaches. We illustrate the performance of our method by solving a challenging benchmark where model-based approaches usually fail and by obtaining promising results in a real-world scenario for controlling a gas turbine.

Subfields: Electrical and Electronic Engineering, Ocean Engineering, Statistical and Nonlinear Physics, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Electrical and Electronic Engineering

In Ocean Engineering

In Statistical and Nonlinear Physics

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Recurrent World Models Facilitate Policy Evolution (2018, background): “Other recent works [12, 17] use Bayesian neural networks instead of GPs to learn a dynamics model.”
World Models (2018, background): “Other recent works (Gal et al., 2016; Depeweg et al., 2016) use Bayesian neural networks instead of GPs to learn a dynamics model.”
Dropout Inference in Bayesian Neural Networks with Alpha-divergences (2017, methodology): “, 2016) and model-based reinforcement learning (Depeweg et al., 2016).”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “Our approach builds upon the wealth of prior work on model-based online RL methods that models the dynamics by Gaussian processes [11], local linear models [41, 37], neural network function approximators [14, 20, 13], and neural video prediction models [15, 31].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Neural network predictive models (Draeger et al., 1995; Gal et al., 2016; Depeweg et al., 2016; Nagabandi et al., 2018), are appealing because they allow for algorithms that combine the sample efﬁciency of a model-based approach with the asymptotic performance of high-capacity function…”
Model-Ensemble Trust-Region Policy Optimization (2018, methodology): “Depeweg et al. (2017) utilize Bayesian neural networks (BNNs) to learn a distribution over dynamics models, and train a policy using gradient-based optimization over a collection of models sampled from this distribution.”
Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion (2018, methodology): “[5] use a Bayesian neural network as the environment model in a policy search setting, learning a policy purely from imagined rollouts.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, background): “Several works approach this problem of model-bias by learning a distribution of models [26, 7, 10, 23], or by learning adaptive models [27, 28, 29].”
Uncertainty in Deep Learning (2016, methodology): “, 2016], which was used in reinforcement learning [Depeweg et al., 2016]).”
A benchmark environment motivated by industrial control problems (2017, methodology): “Example instances of these settings can be found in [18] and [10].”
Model-Based Reinforcement Learning for Sepsis Treatment (2018, methodology): “The description provided here is based on that in Depeweg et al. [2].”
Few-shot Language Coordination by Modeling Theory of Mind (2021, methodology): “Recently, neural networks have been used widely for model-based RL (Gal et al., 2016; Depeweg et al., 2016; Nagabandi et al., 2018; Chua et al., 2018; Janner et al., 2019).”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “(Depeweg et al., 2017; Gal et al., 2016b; Killian et al., 2017)): here, it is essential not only to have an estimate of what state an agent may be in after taking a particular action, but also an accurate sense of all the states the agent may end up in.”
Direct Policy Transfer via Hidden Parameter Markov Decision Processes (2018, methodology): “Initially the policy acts in an -greedy 1An alternative would be to use input-uncertainty BNNs, as done by Depeweg et al. (2016). fashion with respect to π(s, wb;Y).”
Conditional Density Estimation with Bayesian Normalising Flows (2018, methodology): “For example, in reinforcement learning, we often want to model action-value functions for risky decisions for which rewards are inherently bimodal, or state transition dynamics which may vary sig-niﬁcantly from Gaussian (Depeweg et al., 2017).”
Uncertainty Estimation in Bayesian Neural Networks And Links to Interpretability (2018, methodology): “5, which is shown to yield improved predictions over α = 0 (standard variational inference objective) and α = 1 ( expectation propagation objective) [31, 15].”