Off-Policy Multi-Agent Decomposed Policy Gradients

By Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, Chongjie Zhang. 2020

220 citations recorded by the source corpus.

Multi-agent policy gradient (MAPG) methods recently witness vigorous progress. However, there is a significant performance discrepancy between MAPG methods and state-of-the-art multi-agent value-based approaches. In this paper, we investigate causes that hinder the performance of MAPG algorithms and present a multi-agent decomposed policy gradient method (DOP). This method introduces the idea of value function decomposition into the multi-agent actor-critic framework. Based on this idea, DOP supports efficient off-policy learning and addresses the issue of centralized-decentralized mismatch and credit assignment in both discrete and continuous action spaces. We formally show that DOP critics have sufficient representational capability to guarantee convergence. In addition, empirical evaluations on the StarCraft II micromanagement benchmark and multi-agent particle environments demonstrate that DOP significantly outperforms both state-of-the-art value-based and policy-based multi-agent reinforcement learning algorithms. Demonstrative videos are available at this https URL.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “COMA [20], MADDPG [9], PR2 [27], and DOP [10] study the problem of policy-based multi-agent reinforcement learning.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, methodology): “However, learning roles from scratch might not be easier than learning without roles, as directly ﬁnding an optimal decomposition suffers from the same problem as other CTDE learning methods – searching in the large joint space with substantial exploration (Wang et al., 2020c).”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, methodology): “CTDE can resolve the non-stationarity under partial observability and has demonstrated great potential to address complex real-world problems, such as trafﬁc signal control (Zhang et al., 2020; Xu et al., 2021) and autonomous driving cars (Zhou et al., 2020).”
FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, methodology): “Results on SMAC show that FACMAC significantly outperforms stochastic DOP [43], which recently claimed to be the first multi-agent actor-critic method to outperform state-of-the-art valued-based methods on SMAC, in all scenarios we tested.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “The datasets are made by collecting DOP [55] training data.”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “The weight w i can either be a fixed value [27, 29] or a learnable parameter subject to certain constraints [30].”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “Unfortunately, the suboptimality of one agent’s policy can propagate through the centralized value function and negatively affect policy learning of other agents, which is named centralized-decentralized mismatch (CDM) [36].”