The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

By Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre M. Bayen, Yi Wu. Neural Information Processing Systems, 2021

2,481 citations recorded by the source corpus.

Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO's empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at \url{https://github.com/marlbenchmark/on-policy}.

Subfields: Artificial Intelligence, Economics and Econometrics, Management Science and Operations Research, Reinforcement Learning

View Paper: https://arxiv.org/abs/2103.01955

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “MAPPO: MAPPO achieves high returns in the vast majority of tasks and only performs slightly worse than other algorithms in some MPE and LBF tasks.”
ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind (2021, methodology): “We adopt standard A2C (Mnih et al., 2016) as the RL training algorithm, while any MARL method with CTDE framework is also applicable, such as PPO (Schulman et al., 2017; Yu et al., 2021).”
Why Do Multi-Agent LLM Systems Fail? (2025, neutral): “MAPPO (Yu et al., 2022) optimizes agents’ adherence to defined roles.”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “To encompass the episodic concept in inﬁnite MDPs, episodic RL tasks usually assume the existence of a set of absorbing states S 0 , which indicates the termination of episodic tasks [18], [19], and any action taken upon an absorbing state will only transit to itself with zero rewards.”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “We find three key techniques: (1) advantage normalization (Raffin et al., 2021), (2) large-batch-size training (Yu et al., 2022), and (3) updating the parameters of the reference model with exponential moving average (Ouyang et al., 2022).”