Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
By Ryan J. Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch. 2017
6,086 citations recorded by the source corpus.
We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.
Subfields: Artificial Intelligence, Computer Science Applications, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Pieter Abbeel
- Ruslan Salakhutdinov
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
In Computer Science Applications
- Shimon Whiteson
- Stefano V. Albrecht
- Filippos Christianos
- Chongjie Zhang
- Wendelin Böhmer
- Georgios Papoudakis
- Christian Schröder de Witt
- Tonghan Wang
In LLM Agents & Tool Use
- Shimon Whiteson
- Zongqing Lu
- Chongjie Zhang
- Jiechuan Jiang
- Wendelin Böhmer
- Richard Ryan Williams
- Jakob N. Foerster
- Stefano V. Albrecht
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Shimon Whiteson
- Chelsea Finn
- Yuxi Li
- Jakob N. Foerster
- Shixiang Shane Gu
- Chongjie Zhang
Citations in the corpus
- AI safety via debate (2018, background): “However, unrestricted dialog models remain far from human performance and even evaluation is difficult [Lowe et al., 2017a], so it is instructive to consider debates with non-natural language statements.”
- Counterfactual Multi-Agent Policy Gradients (2017, background): “(Lowe et al. 2017) concurrently propose a multi-agent policy-gradient algorithm using centralised critics.”
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “[21] use a concatenation of local observations (CL) global state which is formed by concatenating all local agent observations.”
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Similarly, Lowe et al. (2017) present MADDPG, which learns a centralised critic for each agent and apply this to competitive games with continuous action spaces.”
- Challenges for Transparency (2017, background): “Recent work explores multi-agent reinforcement learning in repeated social interactions to begin to identify conditions on the environment and the agents’ cognitive capacities which lead to the emergence of cooperation (see Leibo et al., 2017; Lowe et al., 2017 and related blog entries).”
- Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “[5,7,13] use a centralised critic to evaluate the action-value of the joint actions.”
- Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “inforcement Learning [5, 38] in cooperative settings has been widely studied [11, 12, 15, 23, 28, 37], and applied to domains such as video games [31] and visual question answering [9].”
- Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, methodology): “At the same time, the framework of centralised learning and decentralised execution was also adopted by MADDPG in (Lowe et al. 2017) in some simpler, non-startcraft cases.”
- Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning (2017, background): “The more recent resurgence has focused on deep RL [1, 8, 10, 13, 18, 19, 23, 26].”
- TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, neutral): “In this experiment, we use QMIX (Rashid et al., 2018), MADDPG (Lowe et al., 2017), CDS (Li et al., 2021) and MAPPO (Yu et al., 2021) as the baselines.”
- Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “COMA [20], MADDPG [9], PR2 [27], and DOP [10] study the problem of policy-based multi-agent reinforcement learning.”
- Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “…al., 2016; Li et al., 2020; Yu et al., 2019), φ is often about the distance between the robot/object and its target position; in general multi-agent games (Lowe et al., 2017; Leibo et al., 2017; Baker et al., 2020), φ could contain each agent’s individual reward as well as the joint reward over…”
- Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, methodology): “This approach is common in the literature and usually encountered when there is no mention of parameter sharing, e.g. MADDPG (Lowe et al., 2017).”
- RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “COMA (Foerster et al., 2018) and MADDPG (Lowe et al., 2017) propose the paradigm of centralized critic with decentralized actors (CCDA).”
- Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “MADDPG and MASAC’s critic shares the same architecture as ˆ Q ∗ .”
- Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “A common assumption of DDPG (and thus MADDPG) is differentiability of actions with respect to the parameters of the actor, so the action space must be continuous.”
- Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, background): “There have been a large number of papers in this space, with seminal work including MADDPG (Lowe et al., 2017) and COMA (Foerster et al., 2018a), both of which are actor-critic methods that employ a centralized critic with decentralized actors.”
- Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[33] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch.”
- The StarCraft Multi-Agent Challenge (2019, methodology): “ting environment dynamics beyond the simple 2D physics nor good reward signals, thus reducing the impact of the environment as a testbed. Multiple gridworld-like environments have also been explored. Lowe et al. (2017) released a set of simple grid-world like environments for multi-agent RL alongsid”