Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control

By Sai Qian Zhang, Qi Zhang, Jieyu Lin. 2019

134 citations recorded by the source corpus.

Multi-agent reinforcement learning (MARL) has recently received considerable attention due to its applicability to a wide range of real-world applications. However, achieving efficient communication among agents has always been an overarching problem in MARL. In this work, we propose Variance Based Control (VBC), a simple yet efficient technique to improve communication efficiency in MARL. By limiting the variance of the exchanged messages between agents during the training phase, the noisy component in the messages can be eliminated effectively, while the useful part can be preserved and utilized by the agents for better performance. Our evaluation using multiple MARL benchmarks indicates that our method achieves $2-10\times$ lower in communication overhead than state-of-the-art MARL algorithms, while allowing agents to achieve better overall performance.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Zhang et al. (2019) also allow for each agent’s Q -values to condition on vector-valued messages from other agents.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “Since their introduction, SMAC and PyMARL have been used by a number of papers to evaluate and compare multi-agent RL methods (Schroeder de Witt et al., 2018; Wang et al., 2019c; Du et al., 2019; Wang et al., 2019a; Zhang et al., 2019; Mahajan et al., 2019; Yao et al., 2019; Bhmer et al., 2019; Wang”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “…[50]; ATOC [39]; DGN [81]; TarMAC [89]; IC3Net [49]; VBC [31]; NDQ [75]; GA-Comm [83]; LSC [77]; Diff Discrete[78]; I2C [79]; ETCNet [86]; Variable-length Coding [87]; [32]; HAMMER [91]; MAGIC [92]; FlowComm [93]; MAIC [33]; FCMNet [94]; Concurrent MD-MADDPG [42]; IS [90]; • Independent Policies .”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, methodology): “To solve this problem, several methods choose to communicate self-state information with other agents [8, 41].”
Succinct and Robust Multi-Agent Communication With Temporal Message Control (2020, methodology): “Recent work [12, 13, 39] have demonstrated that allowing inter-agent communication during execution can greatly enhance the overall performance.”