Decentralized Stochastic Control with Partial History Sharing: A Common Information Approach

By A. Nayyar, Aditya Mahajan, D. Teneketzis. 2012

313 citations recorded by the source corpus.

A general model of decentralized stochastic control called partial history sharing information structure is presented. In this model, at each step the controllers share part of their observation and control history with each other. This general model subsumes several existing models of information sharing as special cases. Based on the information commonly known to all the controllers, the decentralized problem is reformulated as an equivalent centralized problem from the perspective of a coordinator. The coordinator knows the common information and selects prescriptions that map each controller's local information to its control actions. The optimal control problem at the coordinator is shown to be a partially observable Markov decision process (POMDP) which is solved using techniques from Markov decision theory. This approach provides 1) structural results for optimal strategies and 2) a dynamic program for obtaining optimal strategies for all controllers in the original decentralized problem. Thus, this approach unifies the various ad-hoc approaches taken in the literature. In addition, the structural results on optimal control strategies obtained by the proposed approach cannot be obtained by the existing generic approach (the person-by-person approach) for obtaining structural results in decentralized problems; and the dynamic program obtained by the proposed approach is simpler than that obtained by the existing generic approach (the designer's approach) for obtaining dynamic programs in decentralized problems.

Subfields: Artificial Intelligence, Computer Networks and Communications, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Control and Systems Engineering

Citations in the corpus

Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, background): “The typical workout for this is a public belief over private features which only conditions on common knowledge and can therefore be calculated by all agents individually, we refer to Morav ˇ cík et al. (2017); Nayyar et al. (2013); Foerster et al. (2018b) for more details.”
Cooperative Inverse Reinforcement Learning (2016, background): “Nayyar et al. (2013) shows that a D ec-POMDP can be reduced to a coordination - POMDP .”
Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning (2018, methodology): “Fortunately, in our setting the common knowledge described above makes it possible to compute a public belief, (Nayyar et al., 2013), that makes the recursion of I-POMDPs unnecessary.”
A Concise Introduction to Decentralized POMDPs (2016, background): “Consequently, it is possible to define the value function of this plan-time MDP in a very similar way; see Oliehoek [2010] for details.”
Improving Policies via Search in Cooperative Partially Observable Games (2019, methodology): “…Action Decoder learning method uses a public belief over private features and explores in the space of deterministic partial policies using deep RL (Foerster et al. 2019), which can be regarded as a scalable instantiation of the general ideas presented in (Nayyar, Mahajan, and Teneketzis 2013).”
Rethinking Formal Models of Partially Observable Multiagent Decision Making (2019, methodology): “t situations in the game are interconnected, and, in general, one cannot be computed without the other. The only known method for decomposing imperfect information games is based on public knowledge (Nayyar et al., 2013; Burch et al., 2014). In games like poker, where most of the information created”