Path integrals and symmetry breaking for optimal control theory

By Hilbert J. Kappen. 2005

408 citations recorded by the source corpus.

This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton–Jacobi–Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton–Jacobi equation to the Schrödinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process that can be computed by stochastic integration or by the evaluation of a path integral. It is shown how in the deterministic limit the Pontryagin minimum principle formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as Monte Carlo sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in a number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.

Subfields: Finance, Statistical and Nonlinear Physics, Mathematical Optimization, AI for Physical Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Finance

In Statistical and Nonlinear Physics

In Mathematical Optimization

In AI for Physical Sciences

Citations in the corpus

Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “In particular, this operator has been used by [5] to propose an iterative algorithm converging to the optimal maximum reward policy inspired by the work of [15, 39].”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “1 Optimization problems of this type have been explored in a number of prior works (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008), which are covered in more detail in Section 4.”
Reinforcement learning in robotics: A survey (2013, background): “In recent years, general reinforcement learning has yielded three kinds of policy search approaches that have translated particularly well into the domain of robotics: (i) policy gradients approaches based on likelihood-ratio estimation (Sutton et al, 2000), (ii) policy updates inspired by expectati”
Maximum a Posteriori Policy Optimisation (2018, methodology): “Among these are methods that perform trajectory optimisation using either linearised dynamics (Todorov, 2008; Toussaint, 2009; Levine & Koltun, 2013) or general dynamics as in path integral control (Kappen, 2005; Theodorou et al., 2010).”
Soft Actor-Critic Algorithms and Applications (2018, background): “Optimization problems of this type have been explored in a number of prior works (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “The idea of constraining policy gradient updates was explored earlier through the use of “natural gradient” updates [46] and also by means of the Kullback-Leibler (KL) divergence [48, 88].”
Dynamic policy programming (2010, methodology): “ loss bounds of incremental RL methods and this study appears to be the ﬁrst to prove such a bound for an incremental RL algorithm. The work proposed in this paper has some relation to recent work by Kappen (2005) and Todorov (2006), who formulate a stochastic optimal control problem to ﬁnd a condit”
Latent Space Policies for Hierarchical Reinforcement Learning (2018, background): “A number of prior works have explored how reinforcement learning can be cast in the framework of probabilistic inference (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008; Toussaint, 2009; Peters et al., 2010; Neumann, 2011).”
Approximate MaxEnt Inverse Optimal Control and Its Application for Mental Simulation of Human Interactions (2015, methodology): “In this context, Monte Carlo sampling has been utilized in (Kappen 2005) to approximate the path integral computation, and function approximation of the desirability function has also been explored in (Todorov 2009; Zhong and Todorov 2011).”
Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, methodology): “Although not widely used, it has received some theoretical analysis (Kappen, 2005, 2007; Lee et al., 2019; K. Rawlik, Toussaint, & Vijayakumar, 2013; K. C. Rawlik, 2013).”
Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems (2008, methodology): “Van den Broek, Wiegerinck, and Kappen (2007) considered a multi-agent system with second-order dynamics, linear autonomous dynamics and zero instantaneous costs, and showed that graphical model inference by naive mean field approximation significantly outperformed a greedy inference.”
Learning variable impedance control (2011, methodology): “Recently, an interesting connection between stochastic optimal control and Monte Carlo evaluations of path integrals was made (Kappen 2005b).”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “MaxEnt RL has appealing connections to probabilistic inference [8, 23, 34, 40, 49, 52, 54, 61], prompting a renewed interest in recent years [1, 17, 27].”
Reinforcement learning of motor skills in high dimensions: A path integral approach (2010, background): “…for optimal controls formulated as u t i = P ( τ i ) u ( τ i ) d τ i with the probability P ( τ i ) and local controls u ( τ i ) deﬁned as The path cost ˜ S ( τ i ) is a generalized version of the path cost in [18], [19], which only considered systems with state independent control transition.”
Black-Box Policy Search with Probabilistic Programs (2015, background): “(fn policy [u vs] (argmax (zipmap vs (map (fn [v] (Q u v)) vs )))))) Figure 1: A Canadian traveler problem (CTP) implementation in Anglican.”
Approximate Inference and Stochastic Optimal Control (2010, background): “Unlike previous theoretical work on this issue [13, 31, 12, 18] this reformulation is exact without making further assumptions, though this comes at the cost of a lack of a closed form solution.”