Optimal control as a graphical model inference problem

By Hilbert J. Kappen, Vicenç Gómez, Manfred Opper. 2009

415 citations recorded by the source corpus.

We reformulate a class of non-linear stochastic optimal control problems introduced by Todorov (in Advances in Neural Information Processing Systems, vol. 19, pp. 1369–1376, 2007) as a Kullback-Leibler (KL) minimization problem. As a result, the optimal control computation reduces to an inference computation and approximate inference methods can be applied to efficiently compute approximate optimal controls. We show how this KL control theory contains the path integral control method as a special case. We provide an example of a block stacking task and a multi-agent cooperative game where we demonstrate how approximate inference can be successfully applied to instances that are too complex for exact computation. We discuss the relation of the KL control approach to other inference approaches to control.

Subfields: Artificial Intelligence, Statistics and Probability, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[1, 33, 48, 61]).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Furthermore, even if RL with KL penalties (Todorov, 2006a; Kappen et al., 2012; Jaques et al., 2017; 2019), the method used to fine-tune a LM in RLHF, is defined only in terms of reward maximization, it has also been shown to be equivalent to minimizing the reverse KL divergence KL(πθ||p) of πθ to a”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Ziegler instantiates the KL-control approach: its objective includes a KL penalty term for departures from a.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “The approach we propose is based on KL-control, a branch of stochastic optimal control (SOC) (Sten-gel, 1986) where the Kullback-Leibler (KL) divergence from some distribution is used to regularize an RL policy (Abdolmaleki et al., 2018; Kappen et al., 2012; Rawlik et al., 2012; Todorov, 2007).”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “…inference and control has been explored in the literature under a variety of names, including the Kalman duality (Todorov, 2008), maximum entropy reinforcement learning (Ziebart, 2010), KLdivergence control (Kappen et al., 2012; Kappen, 2011), and stochastic optimal control (Toussaint, 2009).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Even though entropy regularized policy optimization is a well studied topic in RL [46, 39, 40, 47, 5, 4, 6, 7]–in fact, one that has been attracting renewed interest from concurrent work [25, 11]–we contribute new observations to this study that are essential for the methods we propose: first, we id”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “…is typically done by augmenting the actor loss with a penalty measuring the divergence of the learned policy from the behavior policy (Jaques et al., 2019; Kumar et al., 2019; Wu et al., 2019), reminiscent of KL-control methods in related literature (Kappen et al., 2012; Jaques et al., 2017).”
Distral: Robust multitask reinforcement learning (2017, background): “The distilled policy π 0 can be interpreted as a policy prior, a perspective well-known in the literature on RL as probabilistic inference [28, 13, 21, 8].”
PLATO: Policy learning using adaptive trajectory optimization (2016, background): “The details of this maximum entropy variant of iLQG may be found in prior work [19], [20], [14].”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, background): “The connections between RL and divergence minimization have long been studied in the rich prior literature of control as probabilistic inference [17, 18, 19, 20, 21, 22, 23].”
Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy (2010, methodology): “A steady line of research has attempted to apply efficient approximate inference techniques in non-decision-based probabilistic graphical models to influence diagrams, Markov decision processes, and other decision modeling frameworks (Cooper, 1988; Shachter & Peot, 1992; Jensen et al., 1994; Zhang, ”
A Minimum Relative Entropy Principle for Learning and Acting (2008, methodology): “A I] 1 Keywords: Artificial Intelligence, Minimum Relative Entropy Principle, Bayesian Control Rule, Interaction Sequences, Operation Modes.”
Making Sense of Reinforcement Learning and Probabilistic Inference (2020, background): “In both cases the connections provide new insights, inspire new algorithms and enrich our understanding (Toussaint & Storkey, 2006; Ziebart et al., 2008; Kappen et al., 2012).”
Optimal control theory and the linear bellman equation (2011, methodology): “The result for t > t1 is a simple linear quadratic control problem for which the solution is given by Eq. 1.38 and for t Kappen (2005b) :”
The anatomy of choice: active inference and agency (2013, result): “This value has exactly the same form as the objective functions in KL control schemes that grandfather conventional utility-based schemes (Kappen et al., 2012; Ortega and Braun, 2011).”