Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

By Bart van den Broek, Wim Wiegerinck, Hilbert J. Kappen. 2008

94 citations recorded by the source corpus.

In this article we consider the issue of optimal control in collaborative multi-agent systems with stochastic dynamics. The agents have a joint task in which they have to reach a number of target states. The dynamics of the agents contains additive control and additive noise, and the autonomous part factorizes over the agents. Full observation of the global state is assumed. The goal is to minimize the accumulated joint cost, which consists of integrated instantaneous costs and a joint end cost. The joint end cost expresses the joint task of the agents. The instantaneous costs are quadratic in the control and factorize over the agents. The optimal control is given as a weighted linear combination of single-agent to single-target controls. The single-agent to single-target controls are expressed in terms of diffusion processes. These controls, when not closed form expressions, are formulated in terms of path integrals, which are calculated approximately by Metropolis-Hastings sampling. The weights in the control are interpreted as marginals of a joint distribution over agent to target assignments. The structure of the latter is represented by a graphical model, and the marginals are obtained by graphical model inference. Exact inference of the graphical model will break down in large systems, and so approximate inference methods are needed. We use naive mean field approximation and belief propagation to approximate the optimal control in systems with linear dynamics. We compare the approximate inference methods with the exact solution, and we show that they can accurately compute the optimal control. Finally, we demonstrate the control method in multi-agent systems with nonlinear dynamics consisting of up to 80 agents that have to reach an equal number of target states.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Learning variable impedance control (2011, background): “In [38], the path integral formalism is extended for stochastic optimal con-”
Learning Policy Improvements with Path Integrals (2010, methodology): “Previous work in [14, 5, 15, 6] are special cases of our generalized formulation.”
A Generalized Path Integral Control Approach to Reinforcement Learning (2010, methodology): “• With respect to the work in Broek et al. (2008), Wiegerinck et al.”
Reinforcement learning of motor skills in high dimensions: A path integral approach (2010, methodology): “…as: The local control can now be expressed as: By substituting T in the equation above we get our main result for the local controls of the sampled path for the generalized path integral formulation: $ % Previous work in [20], [18], [19], [11] are special cases of our generalized formulation.”
Black-Box Policy Search with Probabilistic Programs (2015, methodology): “cy search as expectation maximization in probabilistic programs ﬁts into a long history of approaches that frame planning and control problems as inference. Path integral methods [Kappen, 2005, 2007, Broek et al., 2008, Todorov, 2009a,b, Kappen et al., 2012] express the expected cost-to-go as a KL-d”
Optimal control theory and the linear bellman equation (2011, methodology): “One can therefore consider various well-known methods from the machine learning community to approximate this path integral, such as the Laplace approximation and Monte Carlo sampling Kappen (2005b), variational approximations or belief propagation Broek et al. (2008a). In section 1.7.2 we show an e”