Stick-Breaking Policy Learning in Dec-POMDPs

By Miao Liu, Chris Amato, Xuejun Liao, Lawrence Carin, Jonathan P. How. 2015

31 citations recorded by the source corpus.

Expectation maximization (EM) has recently been shown to be an efficient algorithm for learning finite-state controllers (FSCs) in large decentralized POMDPs (Dec-POMDPs). However, current methods use fixed-size FSCs and often converge to maxima that are far from the optimal value. This paper represents the local policy of each agent using variable-sized FSCs that are constructed using a stick-breaking prior, leading to a new framework called decentralized stick-breaking policy representation (Dec-SBPR). This approach learns the controller parameters with a variational Bayesian algorithm without having to assume that the Dec-POMDP model is available. The performance of Dec-SBPR is demonstrated on several benchmark problems, showing that the algorithm scales to large problems while outperforming other state-of-the-art methods.

Subfields: Control and Systems Engineering, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

1. Zhuoran Yang
2. Jonathan P. How
3. Tamer Basar
4. Jason Pazis
5. Chris Amato
6. John L. Vian
7. Miao Liu
8. K. Åström

In Electrical and Electronic Engineering

1. Zhuoran Yang
2. Jonathan P. How
3. Kaiqing Zhang
4. Tamer Basar
5. Chris Amato
6. Miao Liu

Citations in the corpus

• Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, methodology): “Recent scalable methods use Expectation Maximization to learn finite state controller (FSC) policies (Wu et al., 2013; Liu et al., 2015; 2016).”
• Learning for multi-robot cooperation in partially observable stochastic environments with macro-actions (2017, unknown): “POMDPs [19], [20] and MacDec-POMDPs [12].”
• Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments (2016, methodology): “To this end, EM has been adapted to model-free RL settings to optimize FSCs (Wu, Zilberstein, and Jennings 2013; Liu et al. 2015).”
• Interactive POMDPs with finite-state models of other agents (2017, methodology): “Moreover, stick-breaking priors have been used over the space of policies for decentralized POMDPs [37].”
• Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, methodology): “A Bayesian nonparametric approach is proposed in [217] to determine the controller size of variable-size FSCs.”