A survey and critique of multiagent deep reinforcement learning

By Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor. 2018

732 citations recorded by the source corpus.

Deep reinforcement learning (RL) has achieved outstanding results in recent years. This has led to a dramatic increase in the number of applications and methods. Recent works have explored learning beyond single-agent scenarios and have considered multiagent learning (MAL) scenarios. Initial results report successes in complex multiagent domains, although there are several challenges to be addressed. The primary goal of this article is to provide a clear overview of current multiagent deep reinforcement learning (MDRL) literature. Additionally, we complement the overview with a broader analysis: (i) we revisit previous key components, originally presented in MAL and RL, and highlight how they have been adapted to multiagent deep reinforcement learning settings. (ii) We provide general guidelines to new practitioners in the area: describing lessons learned from MDRL works, pointing to recent benchmarks, and outlining open avenues of research. (iii) We take a more critical tone raising practical challenges of MDRL (e.g., implementation and computational demands). We expect this article will help unify and motivate future research to take advantage of the abundant literature that exists (e.g., RL and MAL) in a joint effort to promote fruitful research in the multiagent community.

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “See Hernandez-Leal, Kartal, and Taylor (2019) for a comprehensive survey on MARL.”
Task Agnostic Continual Learning via Meta Learning (2019, background): “It has been observed that such non-stationarity in multi-agent systems usually causes catastrophic forgetting of the agent (Hernandez-Leal et al., 2019).”
"Other-Play" for Zero-Shot Coordination (2020, methodology): “Though there is a large literature focusing on various issues in multi-agent optimization (Busoniu et al., 2006; Hernandez-Leal et al., 2019), our paper is agnostic to the precise method used.”
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review (2020, background): “…has two advantages: ① the strong feature extraction ability of DNN helps avoid the manually feature design process, and the control decisions can be directly derived from the raw in‐ puts through end-to-end learning procedure; ② DNN helps RL generalize problems with a large state space [24].”
A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “1 In cooperative scenarios, Hernandez-Leal et al. [16] use learning communication to denote the area of learning communication protocols to promote the cooperation of agents.”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, methodology): “This is also referred to as the combinatorial nature of MARL [20].”
Explainable AI Methods - A Brief Overview (2020, methodology): “Search and construction of decision rules is done by reinforcement learning (RL) [81], [32] in combination with a modiﬁed beam search, a heuristic search algorithm that extends the most promising nodes in a graph.”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “1 INTRODUCTION In recent years, cooperative multi-agent reinforcement learning is an active research area [4, 5, 14, 40] and has made exciting”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, methodology): “We refer the readers to [85], [87], and [88] for a extensive review of deep MARL literature.”