Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks

By Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, Stefano V. Albrecht. 2020

367 citations recorded by the source corpus.

Multi-agent deep reinforcement learning (MARL) suffers from a lack of commonly-used evaluation tasks and criteria, making comparisons between approaches difficult. In this work, we provide a systematic evaluation and comparison of three different classes of MARL algorithms (independent learning, centralised multi-agent policy gradient, value decomposition) in a diverse range of cooperative multi-agent learning tasks. Our experiments serve as a reference for the expected performance of algorithms across different learning tasks, and we provide insights regarding the effectiveness of different learning approaches. We open-source EPyMARL, which extends the PyMARL codebase to include additional algorithms and allow for flexible configuration of algorithm implementation details such as parameter sharing. Finally, we open-source two environments for multi-agent research which focus on coordination under sparse rewards.

Subfields: Artificial Intelligence, Computer Science Applications, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Wang
2. Chao Yu
3. Alexandre M. Bayen
4. Eugene Vinitsky
5. Akash Velu
6. Yi Wu
7. Sergey Levine

In Computer Science Applications

1. Yi Wu

In Reinforcement Learning

1. Yu Wang
2. Alexandre M. Bayen
3. Chao Yu
4. Eugene Vinitsky
5. Akash Velu
6. Yi Wu
7. Sergey Levine

In Evaluation & Benchmarks

1. Eugene Vinitsky
2. Alexandre M. Bayen
3. Yu Wang

Citations in the corpus

• The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “[25] perform a broad benchmark of various MARL algorithms and note that PPO-based methods often perform competitively to other methods.”
• A survey of multi-agent deep reinforcement learning with communication (2022, neutral): “These settings can make it challenging to fairly compare the relative strengths and limitations of algorithms based on their performance outcomes [111].”