New inference strategies for solving Markov Decision Processes using reversible jump MCMC

By Matt Hoffman, Hendrik Kück, Nando de Freitas, Arnaud Doucet. 2009

36 citations recorded by the source corpus.

In this paper we build on previous work which uses inferences techniques, in particular Markov Chain Monte Carlo (MCMC) methods, to solve parameterized control problems. We propose a number of modifications in order to make this approach more practical in general, higher-dimensional spaces. We first introduce a new target distribution which is able to incorporate more reward information from sampled trajectories. We also show how to break strong correlations between the policy parameters and sampled trajectories in order to sample more freely. Finally, we show how to incorporate these techniques in a principled manner to obtain estimates of the optimal policy.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “We are therefore considering the finite horizon setting that is popular in AB tests Kohavi et al. [2009], Scott [2010] and is often studied under the umbrella of best arm identification in the bandits literature [Bubeck et al.”
Portfolio Allocation for Bayesian Optimization (2010, background): “further details we refer the reader to [18].”
Expectation-maximization Methods for Solving (po)mdps and Optimal Control Problems (2010, methodology): “Hoffman et al. (2008, 2009b) uses MCMC methods for approximate inference in this context and generalize the EM algorithm for continuous MDPs (Hoffman et al., 2009a).”
Bayesian Multi-Scale Optimistic Optimization (2014, methodology): “(Approaches within the framework of Bayesian nonlinear experimental design, such as (Hennig & Schuler, 2012) for finding maxima and (Kueck et al., 2006, 2009; Hoffman et al., 2009) for learning functions and Markov decision processes, have to rely on expensive approximate inference for computing int”
An Entropy Search Portfolio for Bayesian Optimization (2014, methodology): “Our ﬁnal experiment compares all 7 methods on a nine-dimensional control problem considered in previous work on Bayesian optimisation portfolios [13, 14].”
Learning to Learn without Gradient Descent by Gradient Descent (2016, methodology): “We also consider an application to a simple reinforcement learning task described by (Hoffman et al., 2009).”