Coordinating multi-agent reinforcement learning with limited communication

By Chongjie Zhang, Victor R. Lesser. 2013

155 citations recorded by the source corpus.

Coordinated multi-agent reinforcement learning (MARL) provides a promising approach to scaling learning in large cooperative multi-agent systems. Distributed constraint optimization (DCOP) techniques have been used to coordinate action selection among agents during both the learning phase and the policy execution phase (if learning is off-line) to ensure good overall system performance. However, running DCOP algorithms for each action selection through the whole system results in significant communication among agents, which is not practical for most applications with limited communication bandwidth. In this paper, we develop a learning approach that generalizes previous coordinated MARL approaches that use DCOP algorithms and enables MARL to be conducted over a spectrum from independent learning (without communication) to fully coordinated learning depending on agents' communication bandwidth. Our approach defines an interaction measure that allows agents to dynamically identify their beneficial coordination set (i.e., whom to coordinate with) in different situations and to trade off its performance and communication cost. By limiting their coordination set, agents dynamically decompose the coordination network in a distributed way, resulting in dramatically reduced communication for DCOP algorithms without significantly affecting overall learning performance. Essentially, our learning approach conducts co-adaptation of agents' policy learning and coordination set identification, which outperforms approaches that sequence them.

Subfields: Artificial Intelligence, Computer Networks and Communications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Reinforcement Learning

Citations in the corpus

Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, background): “The rapid progress in recent years of machine learning, and deep learning in particular, opens the door to a new perspective on this debate.”
Learning Multiagent Communication with Backpropagation (2016, methodology): “Others use co unic ti n, but with a p -d rmin d p otoc l [30, 19, 37, 17].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Cooperative multi-agent problems are ubiquitous in real-world applications, such as crewless aerial vehicles (Pham et al., 2018; Xu et al., 2018) and sensor networks (Zhang & Lesser, 2013).”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “There has been a plethora of work on multi-agent reinforcement learning with communication, e.g., (Tan, 1993; Melo et al., 2011; Panait & Luke, 2005; Zhang & Lesser, 2013; Maravall et al., 2013).”
Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, background): “Foer-ster et al. (2016); Sukhbaatar et al. (2016); Peng et al. (2017); Guestrin et al. (2002), and Zhang & Lesser (2013) train multiple agents to learn a communication protocol, and have shown that communicating agents achieve better rewards at various tasks.”
A Multi-Agent Deep Reinforcement Learning Method for Cooperative Load Frequency Control of a Multi-Area Power System (2020, background): “With appropriate training protocol and reward function, data-driven control schemes based on multi-agent reinforcement learning (RL) can obtain cooperative behaviour without full communication between the controllers [21].”
Cooperation and communication in multiagent deep reinforcement learning (2016, background): “There is an extensive body of literature on communication between reinforcement learning agents (Tan, 1998; Panait and Luke, 2005; Zhang and Lesser, 2013).”
Hierarchical Deep Multiagent Reinforcement Learning (2018, background): “However, many real-world tasks such as sensor networks (Zhang and Lesser 2013), trafﬁc control (Van der Pol and Oliehoek 2016), are naturally modeled as multiagent systems (MASs), where multiple agents have cooperative or competitive interactions with others.”
Who2com: Collaborative Perception via Learnable Handshake Communication (2020, background): “Early attempts utilized a pre-defined communication protocol to propagate the information across agents [37, 39, 27], while dynamics of environments and a varied number of agents gave rise to the development of learnable mulit-agent communication [36, 11, 2, 14, 10, 29, 17, 35, 18, 7, 30, 9, 15, 19]”