Learning Nearly Decomposable Value Functions Via Communication Minimization

By Tonghan Wang, Jianhao Wang, Chongyi Zheng, Chongjie Zhang. 2019

179 citations recorded by the source corpus.

Reinforcement learning encounters major challenges in multi-agent settings, such as scalability and non-stationarity. Recently, value function factorization learning emerges as a promising way to address these challenges in collaborative multi-agent systems. However, existing methods have been focusing on learning fully decentralized value functions, which are not efficient for tasks requiring communication. To address this limitation, this paper presents a novel framework for learning nearly decomposable Q-functions (NDQ) via communication minimization, with which agents act on their own most of the time but occasionally send messages to other agents in order for effective coordination. This framework hybridizes value function factorization learning and communication learning by introducing two information-theoretic regularizers. These regularizers are maximizing mutual information between agents' action selection and communication messages while minimizing the entropy of messages between agents. We show how to optimize these regularizers in a way that is easily integrated with existing value function factorization methods such as QMIX. Finally, we demonstrate that, on the StarCraft unit micromanagement benchmark, our framework significantly outperforms baseline methods and allows us to cut off more than $80\%$ of communication without sacrificing the performance. The videos of our experiments are available at this https URL.

Subfields: Artificial Intelligence, Computer Networks and Communications, Sociology and Political Science, Reinforcement Learning, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Sociology and Political Science

In Reinforcement Learning

In Efficiency & Compression

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Wang et al. (2019a) extend QMIX to a setting which allows for communication between agents.”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “Enjoying these advantages, many advanced deep MARL approaches adopt the PDSP paradigm, including value-based methods [4–8], policy gradients [9–13] and communication learning algorithms [14, 15].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “This sigmoid activation of λi brings sparsity to the credit assignment of the joint advantage function to individuals, which enables efficient multi-agent learning [19].”
Multi-agent deep reinforcement learning: a survey (2021, background): “…by applying intrinsic motivation (Jaques et al. 2018, 2019), by making the communication protocol more robust or scalable (Kim et al. 2019; Singh et al. 2019), and maximizing the utility of the communication through efficient encoding (Celikyilmaz et al. 2018; Li et al. 2019b; Wang et al. 2020c).”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “…[73]; RIAL [73]; CommNet [48]; GCL [61]; BiCNet [50]; IC [62]; TarMAC [89]; MADDPG-M [41]; IC3Net [49]; Sched-Net [43]; DCC-MD [82]; VBC [31]; NDQ [75]; Bias [63]; Diff Discrete[78]; I2C [79]; IS [90]; ETCNet [86]; Variable-length Coding [87]; TMC [32]; AE-Comm [65]; DCSS [64]; R-MACRL [95]; MAIC…”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, methodology): “Advances in valued-based MARL (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Wang et al., 2020e) contribute signiﬁcantly to the progress, achieving state-of-the-art performance on challenging tasks, such as StarCraft II micromanagement (Samvelyan et al., 2019).”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, background): “[16] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang.”