Planning with macro-actions in decentralized POMDPs

By Chris Amato, George Dimitri Konidaris, Leslie Pack Kaelbling. 2014

88 citations recorded by the source corpus.

Decentralized partially observable Markov decision processes (Dec-POMDPs) are general models for decentralized decision making under uncertainty. However, they typically model a problem at a low level of granularity, where each agent's actions are primitive operations lasting exactly one time step. We address the case where each agent has macro-actions: temporally extended actions which may require different amounts of time to execute. We model macro-actions as 'options' in a factored Dec-POMDP model, focusing on options which depend only on information available to an individual agent while executing. This enables us to model systems where coordination decisions only occur at the level of deciding which macro-actions to execute, and the macro-actions themselves can then be executed to completion. The core technical difficulty when using options in a Dec-POMDP is that the options chosen by the agents no longer terminate at the same time. We present extensions of two leading Dec-POMDP algorithms for generating a policy with options and discuss the resulting form of optimality. Our results show that these algorithms retain agent coordination while allowing near-optimal solutions to be generated for significantly longer horizons and larger state-spaces than previous Dec-POMDP methods.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Multi-Agent Common Knowledge Reinforcement Learning (2018, background): “Amato et al. (2014) propose MacDec-POMDPs, which use hierarchically optimal policies that allow agents to under-take temporally extended macro-actions.”
Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments (2016, methodology): “Recent research has addressed the more scalable macro-action based Dec-POMDP (MacDec-POMDP) case where each agent has macro-actions (temporally-extended actions), which may require different amounts of time to execute (Amato, Konidaris, and Kaelbling 2014; Amato et al. 2015a).”
Hierarchical Multiagent Reinforcement Learning for Maritime Traffic Management (2020, methodology): “Our work is motivated by hierarchical reinforcement learning (HRL) [2, 5, 26, 32], a framework for control with temporally extended actions.”
Learning for multi-robot cooperation in partially observable stochastic environments with macro-actions (2017, background): “To combat this issue, recent research has addressed the more scalable macro-action based Dec-POMDP (MacDec-POMDP), where each agent has temporally-extended actions, which may require different amounts of time to complete [7].”