Using Expectation-Maximization for Reinforcement Learning

By Peter S. Dayan, Geoffrey E. Hinton. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, methodology): “A different class of safe and fast policy-search methods, that are inspired by EM, can be derived when the reward is treated as an improper probability distribution (Dayan and Hinton, 1997).”
Human-in-the-Loop Imitation Learning using Remote Teleoperation (2020, background): “(1) can be maximized via Expectation-Maximization [12], which alternates between optimizing the trajectory distribution q while holding the policy ﬁxed and optimizing the policy parameters θ while holding q ﬁxed.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “Casting Reinforcement Learning (RL) as an inference problem has a long history dating back at least two decades (Dayan & Hinton, 1997).”
Optimal control as a graphical model inference problem (2009, positive): “This idea is further developed into an ExpectationMaximization (EM) type algorithm (Dayan and Hinton 1997) in recent work (Peters et al.”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “More related is the work of Peters and Schaal [2007] who generalized the expectation maximization framework of Dayan and Hinton [1997] to convert policy search into (on-policy) weighted regression for the immediate reward setting.”
Particle Value Functions (2017, background): “The idea of treating reinforcement learning as an inference problem is not a new idea (see e.g. Albertini & Runggaldier, 1988; Dayan & Hinton, 1997; Kappen, 2005; Toussaint & Storkey, 2006; Hoffman et al., 2007; Tishby & Polani, 2011; Kappen et al., 2012).”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “MaxEnt RL has appealing connections to probabilistic inference [8, 23, 34, 40, 49, 52, 54, 61], prompting a renewed interest in recent years [1, 17, 27].”
The anatomy of choice: active inference and agency (2013, background): “This formalizes the notion of unconscious inference in perception (Helmholtz, 1866/1962; Dayan et al., 1995; Dayan and Hinton, 1997) and—under some simplifying assumptions—reduces to predictive coding (Rao and Ballard, 1999).”
Non-parametric Policy Search with Limited Information Loss (2017, methodology): “The critical step of choosing the transformation was done by manual design in earlier work (Peters and Schaal, 2007; Dayan and Hinton, 1997)), while here the transformation directly results from the optimization problem (1)–(4).”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, methodology): “Methods that derive from the pseudo-likelihood inference framework [12, 64, 46, 26, 44, 1] and use expectationmaximisation (EM) favour risk-seeking policies [34], which can be suboptimal.”
Robot trajectory optimization using approximate inference (2009, methodology): “relation between expectation-maximization and (immediate reward) RL observed by ( Dayan & Hinton, 1997 ), Attias’ (2003) inference approach to planning, and an EM approach to solving MDPs and POMDPs (Toussaint et al., 2006).”
Towards Motor Skill Learning for Robotics (2007, background): “It turned out [11, 19, 20, 13] that as a new cost function we can maximize the distance D(R(τ)p(τ)||pθ (τ)) between return- or”
Variational methods for Reinforcement Learning (2010, background): “We follow a construction equivalent to (Toussaint et al., 2006) but which has the advantage of not requiring auxiliary variables, see e.g. (Dayan and Hinton, 1997; Kober and Peters, 2009; Furmston and Barber, 2009).”
A Survey on Policy Search for Robotics (2013, unknown): “, p(R|τ ) ∝ R(τ ) − minτ j R(τ j) [19].”