Probabilistic inference for solving discrete and continuous state Markov Decision Processes

By Marc Toussaint, Amos J. Storkey. 2006

552 citations recorded by the source corpus.

Inference in Markov Decision Processes has recently received interest as a means to infer goals of an observed action, policy recognition, and also as a tool to compute policies. A particularly interesting aspect of the approach is that any existing inference technique in DBNs now becomes available for answering behavioral question--including those on continuous, factorial, or hierarchical state representations. Here we present an Expectation Maximization algorithm for computing optimal policies. Unlike previous approaches we can show that this actually optimizes the discounted expected future return for arbitrary reward functions and without assuming an ad hoc finite total time. The algorithm is generic in that any inference technique can be utilized in the E-step. We demonstrate this for exact inference on a discrete maze and Gaussian belief state propagation in continuous stochastic optimal control problems.

Subfields: Artificial Intelligence, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “Note that Toussaint and Storkey (2006) also propose an expectation-maximization based algorithm for control as inference, but in a framework that does in fact yield maximum expected reward solutions, with a similar formulation to the one in this article.”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “Yet, if one would have used more complex optimization leveraging continuous domain planning techniques (e.g. [ 115]), the zones of reachability would be increased if obstacles are introduced since the low-level system could learn to go around them.”
A Distributional Perspective on Reinforcement Learning (2017, methodology): “The planning as inference approach (Toussaint & Storkey, 2006; Hoffman et al., 2009) embeds the return into a graphical model, and applies probabilistic inference to determine the sequence of actions leading to maximal expected reward.”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “There are also conceptual connections to ideas formulated around “RL as inference”, where the solution approach is to infer the values of actions conditioned on the trajectory being optimal [Attias, 2003; Toussaint and Storkey, 2006] (see [Levine, 2018] for a review).”
The anatomy of choice: active inference and agency (2013, background): “…approach adopted in this paper highlights the necessarily intimate relationship between inferring states of the world and optimal behavior (Toussaint and Storkey, 2006; Gläscher et al., 2010), the conﬁdence or precision of that inference (De Martino et al., 2012), and the functional…”
Learning Policy Improvements with Path Integrals (2010, methodology): “In new developments, RL researchers have started to combine the well-developed methods from statistical learning and empirical inference with classical RL approaches in order to minimize tuning parameters and numerical problems, such that ultimately more efficient algorithms can be developed that sc”
Bayesian Time Series Models: Expectation maximisation methods for solving (PO)MDPs and optimal control problems (2011, methodology): “Toussaint and Storkey (2006) ; Toussaint et al. (2006) introduced an alternative based on a mixture of nite-length processes.”
Goal-directed decision making in prefrontal cortex: a computational framework (2008, methodology): “It seems possible that there may be a formal equivalence between the algorithm we have proposed and the one reported by [10].”
A free energy principle for a particular physics (2019, background): “…to homoeostasis (Ashby, 1947; Bernard, 1974) and (ii) deep, long-term, active inference – that speaks to planning and allostasis (Attias, 2003; Botvinick and Toussaint, 2012; Ramsay and Woods, 2014; Stephan et al., 2016; Sterling and Eyer, 1988; Sutton et al., 1999; Toussaint and Storkey, 2006).”
Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. (2012, methodology): “However, as we later discuss in detail, it has precedents in economics, psychology, and neuroscience, as well as in decision theory and machine learning (Shachter & Peot, 1992; Toussaint & Storkey, 2006).”