Variational Bayesian Optimization for Runtime Risk-Sensitive Control
By Scott Kuindersma, R. Grupen, Andrew Barto. 2012
23 citations recorded by the source corpus.
We present a new Bayesian policy search algorithm suitable for problems with policy-dependent cost variance, a property present in many robot control tasks. We extend recent work on variational heteroscedastic Gaussian processes to the optimization case to achieve efficient minimization of very noisy cost signals. In contrast to most policy search algorithms, our method explicitly models the cost variance in regions of low expected cost and permits runtime adjustment of risk sensitivity without relearning. Our experiments with artificial systems and a real mobile manipulator demonstrate that flexible risk-sensitive policies can be learned in very few trials.
Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Emma Brunskill
- Ofir Nachum
- Peter Stone
- Chelsea Finn
- Lihong Li
- Shixiang Shane Gu
In Probabilistic & Bayesian Methods
- Sergey Levine
- Pieter Abbeel
- Teodor Mihai Moldovan
- Emma Brunskill
- Philip S. Thomas
- Bruno Castro da Silva
- Michael I. Jordan
- Andrew Barto
Citations in the corpus
- Model-based reinforcement learning with parametrized physical models and optimism-driven exploration (2015, methodology): “Prior work has proposed to use optimism-driven exploration mainly for discrete systems [4], [19].”
- Optimism-driven exploration for nonlinear systems (2015, methodology): “Although many methods have been proposed for model-based RL with efficient exploration in discrete MDPs, data-efficient model-based RL in continuous systems remains a challenging problem despite substantial recent advances [3], [4], [5].”
- On Ensuring that Intelligent Machines Are Well-Behaved (2017, background): “…is often chosen to be a measure of the expected performance of the robot if it uses the controller θ. Risk-sensitive methods might define f(θ) to also incorporate a term that penalizes solutions, θ, that cause the observed performance of the robot to have high variance [Kuindersma et al., 2012].”