Multi-agent Reinforcement Learning in Sequential Social Dilemmas
By Joel Z. Leibo, Vinícius Flores Zambaldi, Marc Lanctot, Janusz Marecki, Thore Graepel. 2017
712 citations recorded by the source corpus.
Matrix games like Prisoner's Dilemma have guided research on social dilemmas for decades. However, they necessarily treat the choice to cooperate or defect as an atomic action. In real-world social dilemmas these choices are temporally extended. Cooperativeness is a property that applies to policies, not elementary actions. We introduce sequential social dilemmas that share the mixed incentive structure of matrix game social dilemmas but also require agents to learn policies that implement their strategic intentions. We analyze the dynamics of policies learned by multiple self-interested independent learning agents, each using its own deep Q-network, on two Markov games we introduce here: 1. a fruit Gathering game and 2. a Wolfpack hunting game. We characterize how learned behavior in each domain changes as a function of environmental factors including resource abundance. Our experiments show how conflict can emerge from competition over shared resources and shed light on how the sequential nature of real world social dilemmas affects cooperation.
Subfields: Artificial Intelligence, Management Science and Operations Research, Sociology and Political Science, Reinforcement Learning, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Shimon Whiteson
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Percy Liang
In Management Science and Operations Research
- Shimon Whiteson
- Stefano V. Albrecht
- Pieter Abbeel
- Sergey Levine
- Filippos Christianos
- Wendelin Böhmer
- Yuxi Li
- Adam Lerer
In Sociology and Political Science
- Shimon Whiteson
- Alex Peysakhovich
- Adam Lerer
- Yuxi Li
- Thore Graepel
- Adrian Weller
- Joel Z. Leibo
- Karl Tuyls
In Reinforcement Learning
- Shimon Whiteson
- Sergey Levine
- Pieter Abbeel
- Jakob N. Foerster
- Yuxi Li
- Stefano V. Albrecht
- Filippos Christianos
- Wendelin Böhmer
In Game & Decision Theory
- Shimon Whiteson
- Karl Tuyls
- Alex Peysakhovich
- Adam Lerer
- Julien Pérolat
- Thore Graepel
- Yuxi Li
- Marc Lanctot
Citations in the corpus
- OpenSpiel: A Framework for Reinforcement Learning in Games (2019, neutral): “Game Reference(s) Backgammon Wikipedia Breakthrough Wikipedia Bridge bidding Wikipedia Catch [51] and [59, Appendix A] Coin Game [64] Connect Four Wikipedia Cooperative Box-Pushing [70] Chess Wikipedia First-price Sealed-bid Auction Wikipedia Go Wikipedia Goofspiel Wikipedia Hanabi (via HLE) Wikiped”
- Counterfactual Multi-Agent Policy Gradients (2017, methodology): “More recently the same method has been used by Leibo et al. (2017) to study the emergence of collaboration and defection in sequential social dilemmas.”
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “…and Silver, 2016), Pong (Tampuu et al., 2017), Keepaway Soccer (Stone et al., 2005), or simple gridworld-like environments (Lowe et al., 2017; Leibo et al., 2017; Yang et al., 2018; Zheng et al., 2017), there are currently no challenging standard testbeds for centralised training with…”
- Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, methodology): “t (not domain) does not dramatically change the total performance. The agents that we evaluate are listed in the table above. 4.2 Environments We use 2D grid worlds with the same basic functioning as Leibo et al. (2017), but with different tasks we call Switch, Fetch and Checkers. We have observatio”
- Challenges for Transparency (2017, background): “Recent work explores multi-agent reinforcement learning in repeated social interactions to begin to identify conditions on the environment and the agents’ cognitive capacities which lead to the emergence of cooperation (see Leibo et al., 2017; Lowe et al., 2017 and related blog entries).”
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “For example, multi-robot control [20], the discovery of communication and language [29, 8, 24], multiplayer games [27], and the analysis of social dilemmas [17] all operate in a multi-agent domain.”
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, methodology): “To avoid this problem, previous work on deep multi-agent RL has limited the use of experience replay to short, recent buffers (Leibo et al., 2017) or simply disabled replay alto-gether (Foerster et al., 2016).”
- Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “…al., 2020; Yu et al., 2019), φ is often about the distance between the robot/object and its target position; in general multi-agent games (Lowe et al., 2017; Leibo et al., 2017; Baker et al., 2020), φ could contain each agent’s individual reward as well as the joint reward over each team, which…”
- Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[61] Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel.”
- The StarCraft Multi-Agent Challenge (2019, background): “, 2005), or simple gridworld-like environments (Lowe et al., 2017; Leibo et al., 2017; Yang et al., 2018; Zheng et al., 2017).”
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “In practice, nevertheless, IQL commonly serves as a surprisingly strong benchmark even in mixed and competitive games (Tampuu et al., 2017; Leibo et al., 2017).”
- Learning with Opponent-Learning Awareness (2017, background): “As an exception, Leibo et al. (2017) consider mixed multi-agent environments and study the emergence of cooperation and competition as a function of the problem setup and the model parameters.”
- Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “We therefore follow the example of recent publications by conducting evaluations in gridworlds with sufficient complexity to warrant a MA-DRL approach [7, 11, 19].”
- A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “In MARL, several agents interact and learn in an environment simultaneously, either competitively such as in Go [89] and Poker [38, 103, 72], cooperatively such as when learning to communicate [22, 91, 35], or some mix of the two [58, 93, 34].”
- A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Other DRL works of note in MARL investigate the effects of learning and sequential decision making in game theory [38, 60].”
- Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, background): “Previous work on first-person games considers either much simpler games (30, 36, 45, 75), simplified agent interfaces (68), or non-learning systems (50, 69), and previously studied multi-agent domains often consist of discrete-state environments (18, 38, 64), have simplified 2D dynamics (23, 41, 53)”
- A survey and critique of multiagent deep reinforcement learning (2018, background): “[156] meanwhile studied independent DQNs in the context of sequential social dilemmas: a Markov game that satisfies certain inequalities [156].”
- Relational Forward Models for Multi-Agent Learning (2018, positive): “…where multiple learning agents perceive and act in a shared environment, has produced impressive results (Jaderberg et al., 2018; Pachocki et al., 2018; Leibo et al., 2017; Hughes et al., 2018; Peysakhovich & Lerer, 2017a; Lerer & Peysakhovich, 2017; Bansal et al., 2017; Lanctot et al., 2017).”
- Emergent Communication through Negotiation (2018, background): “Recent work on deep multi-agent reinforcement learning (MARL) has shown great success in teaching agents complex behaviour without a complex environment simulator or demonstration data (Panait & Luke, 2005; Busoniu et al., 2008; Tuyls & Weiss, 2012; Silver et al., 2017; Leibo et al., 2017).”