Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning

By Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, Yaodong Yang. International Conference on Learning Representations, 2021

407 citations recorded by the source corpus.

Trust region methods rigorously enabled reinforcement learning (RL) agents to learn monotonically improving policies, leading to superior performance on a variety of tasks. Unfortunately, when it comes to multi-agent reinforcement learning (MARL), the property of monotonic improvement may not simply apply; this is because agents, even in cooperative games, could have conflicting directions of policy updates. As a result, achieving a guaranteed improvement on the joint policy where each agent acts individually remains an open challenge. In this paper, we extend the theory of trust region learning to MARL. Central to our findings are the multi-agent advantage decomposition lemma and the sequential policy update scheme. Based on these, we develop Heterogeneous-Agent Trust Region Policy Optimisation (HATPRO) and Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) algorithms. Unlike many existing MARL algorithms, HATRPO/HAPPO do not need agents to share parameters, nor do they need any restrictive assumptions on decomposibility of the joint value function. Most importantly, we justify in theory the monotonic improvement property of HATRPO/HAPPO. We evaluate the proposed methods on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that HATRPO and HAPPO significantly outperform strong baselines such as IPPO, MAPPO and MADDPG on all tested tasks, therefore establishing a new state of the art.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/27302766f8d0eb6c052eb400e234c5be0e7a767e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Weinan Zhang
2. Jun Wang
3. Yaodong Yang
4. Ying Wen
5. Hongyuan Zha
6. Shangding Gu
7. Yali Du
8. Alois C. Knoll

In Electrical and Electronic Engineering

1. Hongyuan Zha
2. Weinan Zhang
3. Jun Wang
4. Yaodong Yang
5. Ying Wen
6. Arash Bahrammirzaee

Citations in the corpus

• Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, methodology): “To elaborate, we have following proposition proposed by Kuba et al. (2021): Proposition 1 (suboptimal) .”
• A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “However, a consensus network used in this method imposes an extra constraint of parameter sharing among neighbouring agents, which could yield suboptimal solutions [146].”
• Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, methodology): “These methods, however, cannot cover the whole complexity of multi-agent interactions; in fact, some of them are shown to fail in the simplest cooperative task [15].”