Approximate Inference and Stochastic Optimal Control

By Konrad Rawlik, Marc Toussaint, Sethu Vijayakumar. 2010

21 citations recorded by the source corpus.

We propose a novel reformulation of the stochastic optimal control problem as an approximate inference problem, demonstrating, that such a interpretation leads to new practical methods for the original problem. In particular we characterise a novel class of iterative solutions to the stochastic optimal control problem based on a natural relaxation of the exact dual formulation. These theoretical insights are applied to the Reinforcement Learning problem where they lead to new model free, o policy methods for discrete and continuous problems.

Subfields: Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

Citations in the corpus

Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Other works have used related methods for reinforcement learning [31, 32, 33, 34, 35].”
Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, methodology): “…of this in the literature is recasting of standard model-free RL as probabilistic variational inference under the control-as-inference framework, which has lead to significantly improved model-free RL algorithms (Haarnoja, Zhou, Hartikainen, et al., 2018; Levine, 2018; K. Rawlik et al., 2010).”
G-Learning: Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “Figure 2 shows, for each case, the mean absolute error between the learned V t and the optimal V * as a function of the iteration t, for Q-learning, G-learning, Ψ-learning [34] [35] and Double-Q-learning [12].”
Action and Perception as Divergence Minimization (2020, background): “RL as inference (Rawlik et al., 2010) does not maximize mutual information because its target is factorized.”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, background): “These methods do not naturally learn deterministic optimal policies and constraining the variational policies to be deterministic renders inference intractable (Rawlik et al., 2010).”
Hierarchical Relative Entropy Policy Search (2014, unknown): “We show that our Hierarchical REPS can learn versatile solutions while also showing an increased performance in terms of learning speed and quality of the found policy in comparison to the nonhierarchical approach.”