Maintaining cooperation in complex social dilemmas using deep reinforcement learning

By Adam Lerer, Alex Peysakhovich. 2017

172 citations recorded by the source corpus.

Social dilemmas are situations where individuals face a temptation to increase their payoffs at a cost to total welfare. Building artificially intelligent agents that achieve good outcomes in these situations is important because many real world interactions include a tension between selfish interests and the welfare of others. We show how to modify modern reinforcement learning methods to construct agents that act in ways that are simple to understand, nice (begin by cooperating), provokable (try to avoid being exploited), and forgiving (try to return to mutual cooperation). We show both theoretically and experimentally that such agents can maintain cooperation in Markov social dilemmas. Our construction does not require training methods beyond a modification of self-play, thus if an environment is such that good strategies can be constructed in the zero-sum case (eg. Atari) then we can construct agents that solve social dilemmas in this environment.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

Citations in the corpus

Learning with Opponent-Learning Awareness (2017, background): “The ‘Coin Game’ was ﬁrst proposed in Lerer and Peysakhovich (2017) as a higher dimensional expansion of the iterated prionser’s dilemma with multi-step actions.”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, background): “One approach to mitigating this issue would be to learn the impact of uncooperative strategies across multiple rounds of negotiation (Lerer and Peysakhovich, 2017).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “r competitive behaviors exist not only as discrete (atomic) actions, but they are temporally extended (over policies).In the related setting of one shot Markov social dilemmas, Lerer and Peysakhovich [173] extended the famous Tit-for-Tat (TFT)20 strategy [184] for DRL (using function approximators) ”
"Other-Play" for Zero-Shot Coordination (2020, background): “A central challenge for AI is constructing agents that can coordinate and cooperate with partners they have not seen before (Kleiman-Weiner et al., 2016; Lerer & Peysakhovich, 2017; Carroll et al., 2019; Shum et al., 2019).”
Multi-agent deep reinforcement learning: a survey (2021, background): “…dilemmas can be classified along psychological variables known from human interaction (Lange et al. 2013) such as the gain of individual benefits (Lerer and Peysakhovich 2017), the fear of future consequences (Pérolat et al. 2017), the assessment of the impact on another agent’s behavior (Jaques…”
Inequity aversion improves cooperation in intertemporal social dilemmas (2018, background): “planning [20, 21, 22, 23] to generate cooperation in this setting.”
Backplay: "Man muss immer umkehren" (2018, unknown): “…that standard independent multi-agent learning does not produce agents that are able to cooperate in social dilemmas (Leibo et al., 2017; Lerer & Peysakhovich, 2017; Peysakhovich & Lerer, 2017; Foerster et al., 2017) or risky coordination games (Yoshida et al., 2008; Peysakhovich & Lerer, 2018).”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, background): “Social Dilemma BMaRS (F) is defined to be any other situations other than above 4 ones, which is also known as social dilemmas in [54]:”
Prosocial learning agents solve generalized Stag Hunts better than selfish ones (2017, background): “However, for positive sum games good outcomes are far from guaranteed (Leibo et al. 2017; Lerer and Peysakhovich 2017).”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, background): “The behavior market realized from the action trading helps mitigate greedy behavior (like the tit-fortat game-theoretic strategy proposed in [108]), enable agents”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, background): “There has also been progress for mixed cooperative/competitive environments: using a centralized critic [57], learning to negotiate [18], anticipating/learning opponent responses in social dilemmas [23, 53], and control in realistic physical environments [3, 7].”
Consequentialist conditional cooperation in social dilemmas with imperfect information (2017, methodology): “Recent work has considered expanding TFT to more complex Markov games either as a heuristic, by learning cooperative and selfish policies and switching between them as needed (Lerer and Peysakhovich 2017), or as an outcome of an endto-end procedure (Foerster et al. 2017c).”
Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning (2020, background): “Most mixed-motive reinforcement learning research has produced policies through self-play [32, 43] or co-training policies in fixed groups [24, 31].”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, background): “As [16, 19] showed, regular reinforcement learning agents in self-play suffer from a drive to greedily maximise their return, which leads to the collapse of the common good and, consequently, low return.”