Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient

By Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, Stuart J. Russell. 2019

348 citations recorded by the source corpus.

Despite the recent advances of deep reinforcement learning (DRL), agents trained by DRL tend to be brittle and sensitive to the training environment, especially in the multi-agent scenarios. In the multi-agent setting, a DRL agent’s policy can easily get stuck in a poor local optima w.r.t. its training partners – the learned policy may be only locally optimal to other agents’ current policies. In this paper, we focus on the problem of training robust DRL agents with continuous actions in the multi-agent learning setting so that the trained agents can still generalize when its opponents’ policies alter. To tackle this problem, we proposed a new algorithm, MiniMax Multi-agent Deep Deterministic Policy Gradient (M3DDPG) with the following contributions: (1) we introduce a minimax extension of the popular multi-agent deep deterministic policy gradient algorithm (MADDPG), for robust policy learning; (2) since the continuous action space leads to computational intractability in our minimax learning objective, we propose Multi-Agent Adversarial Learning (MAAL) to efficiently solve our proposed formulation. We empirically evaluate our M3DDPG algorithm in four mixed cooperative and competitive multi-agent environments and the agents trained by our method significantly outperforms existing baselines.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “…learning when to cooperate/compete (Littman, 2001; Peysakhovich & Lerer, 2018a; Kleiman-Weiner et al., 2016; Woodward et al., 2019; McKee et al., 2020), which is a special case of ours, or learning robust policies (Li et al., 2019; Shen & How, 2019; Hu et al., 2020), which complements our method.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “M3DDPG [190] Extend MADDPG with minimax objective to robustify the learned policy.”
CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning (2018, background): “While MADDPG [19] and M3DDPG [14] apply to agents with different reward, they do not specifically address the need for cooperation; in fact, they do not distinguish the problems of cooperation and competition, despite the fundamental difference.”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, background): “However, work on decentralised continuous control has received comparatively little attention and has been largely limited to deep deterministic policy gradient approaches (Lowe et al. 2017; Iqbal and Sha 2019; Li et al. 2019).”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “Another line of works [51, 34] consider the adversarial setting of multi-agent reinforcement learning [70, 9].”
A survey of multi-agent deep reinforcement learning with communication (2022, positive): “In MARL, agents’ policies can be sensitive to environmental noise or malicious intentions of opponents, and thus robust policies are required [121, 122].”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, background): “, 2018), and how to learn robustly with either malicious/adversarial or failed/dysfunctional learning agents (Gao et al., 2018; Li et al., 2019; Zhang et al., 2019).”
Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning (2019, methodology): “A recent example in this direction is the work by Li et al. (2019) who used an approximate solution of the Minimax equilibrium.”
Multi-agent deep reinforcement learning: a survey (2021, background): “…by applying intrinsic motivation (Jaques et al. 2018, 2019), by making the communication protocol more robust or scalable (Kim et al. 2019; Singh et al. 2019), and maximizing the utility of the communication through efficient encoding (Celikyilmaz et al. 2018; Li et al. 2019b; Wang et al. 2020c).”