Coordinated Multi-Agent Reinforcement Learning in Networked Distributed POMDPs

By Chongjie Zhang, Victor R. Lesser. 2011

137 citations recorded by the source corpus.

In many multi-agent applications such as distributed sensor nets, a network of agents act collaboratively under uncertainty and local interactions. Networked Distributed POMDP (ND-POMDP) provides a framework to model such cooperative multi-agent decision making. Existing work on ND-POMDPs has focused on offline techniques that require accurate models, which are usually costly to obtain in practice. This paper presents a model-free, scalable learning approach that synthesizes multi-agent reinforcement learning (MARL) and distributed constraint optimization (DCOP). By exploiting structured interaction in ND-POMDPs, our approach distributes the learning of the joint policy and employs DCOP techniques to coordinate distributed learning to ensure the global learning performance. Our approach can learn a globally optimal policy for ND-POMDPs with a property called groupwise observability. Experimental results show that, with communication during learning and execution, our approach significantly outperforms the nearly-optimal non-communication policies computed offline.

Subfields: Artificial Intelligence, Computer Networks and Communications, LLM Agents & Tool Use, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In LLM Agents & Tool Use

In Reinforcement Learning

Citations in the corpus

Multi-agent reinforcement learning as a rehearsal for decentralized planning (2016, neutral): “While we have developed the framework of rehearsal based RL in Dec-POMDPs, it can also be applied to POMDPs (i.e., Dec-POMDPs with n¼1 agent).”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “Cooperative multi-agent reinforcement learning (MARL) has drawn increasing interest in recent years, which provides a promise for solving many real-world challenging problems, such as sensor networks [1], traffic management [2], and coordination of robot swarms [3].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Zhang & Lesser (2011); Kok & Vlassis (2006) also utilize the sparsity and locality of coordination.”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “Cooperative multi-agent reinforcement learning (MARL) has broad prospects for addressing many complex real-world problems, such as sensor networks (Zhang & Lesser, 2011), coordination of robot swarms (Hüttenrauch et al., 2017), and autonomous cars (Cao et al., 2012).”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, methodology): “Cooperative multi-agent reinforcement learning (MARL) are ﬁnding applications in many real-world domains, such as autonomous vehicle teams (Cao et al., 2012), intelligent warehouse systems (Now ´ e et al., 2012), and sensor networks (Zhang & Lesser, 2011).”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “Zhang & Lesser (2011) employ the distributed constraint optimization technique to coordinate distributed learning of joint action-value functions.”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, background): “Cooperative multi-agent reinforcement learning (MARL) has great promise for addressing coordination problems in a variety of applications, such as robotic systems [1], autonomous cars [2], and sensor networks [3].”
Sample Bounded Distributed Reinforcement Learning for Decentralized POMDPs (2012, background): “More recently, Zhang and Lesser (2011) have applied distributed reinforcement learning to a variant of the ﬁnite horizon Dec-POMDP problem, where agents are organized in a network, and agents’ inﬂuence on each other are limited to cliques.”
Influence-Based Multi-Agent Exploration (2019, background): “…been focusing on single-agent scenarios and leave the exploration problem largely unstudied for multi-agent settings – it is common in real-world applications that multiple agents are required to solve a task in a coordinated fashion (Cao et al., 2012; Now´e et al., 2012; Zhang & Lesser, 2011).”
Coordinating multi-agent reinforcement learning with limited communication (2013, methodology): “As with [16], in this paper, we use the max-sum algorithm [11] for computing an approximate solution, which requires much less communication and computation and can be readily implemented as an anytime algorithm to trade off the quality and efficiency of computing joint actions.”
Concurrent reinforcement learning as a rehearsal for decentralized planning under uncertainty (2013, unknown): “Traditional Dec-POMDP solvers are limited in that they compute the set of prescribed behaviors for agents centrally, and assume that a comprehensive set of model parameters are available a priori; however, multi-agent reinforcement learning (MARL) techniques, have recently been applied [4, 1] to ove”