A review of cooperative multi-agent deep reinforcement learning

By Afshin Oroojlooyjadid, Davood Hajinezhad. 2019

635 citations recorded by the source corpus.

Deep Reinforcement Learning has made significant progress in multi-agent systems in recent years. The aim of this review article is to provide an overview of recent approaches on Multi-Agent Reinforcement Learning (MARL) algorithms. Our classification of MARL approaches includes five categories for modeling and solving cooperative multi-agent reinforcement learning problems: (I) independent learners, (II) fully observable critics, (III) value function factorization, (IV) consensus, and (IV) learn to communicate. We first discuss each of these methods, their potential challenges, and how these challenges were mitigated in the relevant papers. Additionally, we make connections among different papers in each category if applicable. Next, we cover some new emerging research areas in MARL along with the relevant recent papers. In light of MARL’s recent success in real-world applications, we have dedicated a section to reviewing these applications and articles. This survey also provides a list of available environments for MARL research. Finally, the paper is concluded with proposals on possible research directions.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “For a more comprehensive survey of cooperative MARL, see [22].”
Deep Coordination Graphs (2019, background): “A general overview over cooperative deep MARL can be found in OroojlooyJadid & Hajinezhad (2019).”
Provable Self-Play Algorithms for Competitive Reinforcement Learning (2020, result): “We refer the readers to several recent surveys for these results (see e.g. Bus¸oniu et al., 2010; Nguyen et al., 2018; OroojlooyJadid and Hajinezhad, 2019; Zhang et al., 2019).”
A survey of multi-agent deep reinforcement learning with communication (2022, neutral): “Furthermore, communication protocols, determining when and with whom to communicate, often leverage deep learning models to find the optimal choices that minimize communication overhead and yield more targeted communication.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Oroojlooyjadid and Hajinezhad (2019) reviewed recent works in the cooperative setting while Da Silva and Costa (2019) and Da Silva et al. (2019) focused on knowledge reuse.”
Large Language Model Guided Tree-of-Thought (2023, methodology): “While there are many multi-agent reinforcement learning algorithms (MARL) proposed in the literature [29, 30, 31], in this work we adopt a relatively simple approach which uses a modiﬁed version of the REINFORCE algorithm [32] to train the policy networks of the ToT controller and the prompter…”
Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning (2019, background): “Naturally, this also applies to multi-agent systems (OroojlooyJadid and Hajinezhad 2019; Bear, Kagan, and Rand 2017; Corgnet, Espı́n, and Hernán-González 2015): agents that maintain consistent cognitions about their environments are crucial for achieving effective system-level cooperation.”