Learning to Schedule Communication in Multi-agent Reinforcement Learning

By Daewoo Kim, Sang-chul Moon, David Earl Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, Yung Yi. 2019

257 citations recorded by the source corpus.

Many real-world reinforcement learning tasks require multiple agents to make sequential decisions under the agents' interaction, where well-coordinated actions among the agents are crucial to achieve the target goal better at these tasks. One way to accelerate the coordination effect is to enable multiple agents to communicate with each other in a distributed manner and behave as a group. In this paper, we study a practical scenario when (i) the communication bandwidth is limited and (ii) the agents share the communication medium so that only a restricted number of agents are able to simultaneously use the medium, as in the state-of-the-art wireless networking standards. This calls for a certain form of communication scheduling. In that regard, we propose a multi-agent deep reinforcement learning framework, called SchedNet, in which agents learn how to schedule themselves, how to encode the messages, and how to select actions based on received messages. SchedNet is capable of deciding which agents should be entitled to broadcasting their (encoded) messages, by learning the importance of each agent's partially observed information. We evaluate SchedNet against multiple baselines under two different applications, namely, cooperative communication and navigation, and predator-prey. Our experiments show a non-negligible performance gap between SchedNet and other mechanisms such as the ones without communication and with vanilla scheduling methods, e.g., round robin, ranging from 32% to 43%.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, methodology): “Other related work includes CommNet (Sukhbaatar et al., 2016), DIAL (Foerster et al., 2016), ATOC (Jiang & Lu, 2018), and SCHEDNET (Kim et al., 2019), which exploit inter-agent communication in execution.”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, background): “In particular, QMIX (Rashid et al., 2018) 2019) study action coordination under limited communication channel and thus are also related to our works.”
Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “The agents of VBC, FC and SchedNet adopt a more aggressive policy, where the allied Zealots try to surround and kill the enemy Zealots first, and then attack the enemy Stalkers by collaborating with the allied Stalkers.”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “…et al., 2018; Grover et al., 2018; Kipf et al., 2018) and explicit communication (Sukhbaatar et al., 2016; Hoshen, 2017; Jiang & Lu, 2018; Singh et al., 2019; Das et al., 2019; Singh et al., 2019; Kim et al., 2019) are studied to model the dependence between the decision-making processes of agents.”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “Such a concern on communication protocols has naturally inspired the recent studies on communication-efficient MARL (Chen et al., 2018; Lin et al., 2019; Ren and Haupt, 2019; Kim et al., 2019).”
Succinct and Robust Multi-Agent Communication With Temporal Message Control (2020, methodology): “Compare with VBC and SchedNet, TMC outperforms VBC and SchedNet by 1.3× and 3.7×, respectively.”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “Both MADDPG and its single-agent counterpart, DDPG, have seen widespread application in Comm-MADRL [39, 40, 41, 42, 43].”
Distributed Learning in Wireless Networks: Recent Progress and Future Challenges (2021, background): “The authors in [187] propose a deep MARL framework in”
Who2com: Collaborative Perception via Learnable Handshake Communication (2020, background): “Early attempts utilized a pre-defined communication protocol to propagate the information across agents [37, 39, 27], while dynamics of environments and a varied number of agents gave rise to the development of learnable mulit-agent communication [36, 11, 2, 14, 10, 29, 17, 35, 18, 7, 30, 9, 15, 19]”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “, 2019 [64] 1 AC 0 (G,L) (G,L) (G,G) Lowe et al.”