Consequentialist conditional cooperation in social dilemmas with imperfect information

By Alex Peysakhovich, Adam Lerer. 2017

66 citations recorded by the source corpus.

Social dilemmas, where mutual cooperation can lead to high payoffs but participants face incentives to cheat, are ubiquitous in multi-agent interaction. We wish to construct agents that cooperate with pure cooperators, avoid exploitation by pure defectors, and incentivize cooperation from the rest. However, often the actions taken by a partner are (partially) unobserved or the consequences of individual actions are hard to predict. We show that in a large class of games good strategies can be constructed by conditioning one's behavior solely on outcomes (ie. one's past rewards). We call this consequentialist conditional cooperation. We show how to construct such strategies using deep reinforcement learning techniques and demonstrate, both analytically and experimentally, that they are effective in social dilemmas beyond simple matrix games. We also show the limitations of relying purely on consequences and discuss the need for understanding both the consequences of and the intentions behind an action.

Subfields: Experimental and Cognitive Psychology, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Safety Research

In Sociology and Political Science

Citations in the corpus

Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “We consider two games adapted from Peysakhovich & Lerer (2018b), Monster-Hunt (Fig.”
Relational Forward Models for Multi-Agent Learning (2018, positive): “…where multiple learning agents perceive and act in a shared environment, has produced impressive results (Jaderberg et al., 2018; Pachocki et al., 2018; Leibo et al., 2017; Hughes et al., 2018; Peysakhovich & Lerer, 2017a; Lerer & Peysakhovich, 2017; Bansal et al., 2017; Lanctot et al., 2017).”
Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning (2018, background): “Thus, recent work explores training agents that learn when to cooperate based solely on their own past rewards (Peysakhovich & Lerer, 2017).”
Inequity aversion improves cooperation in intertemporal social dilemmas (2018, background): “planning [20, 21, 22, 23] to generate cooperation in this setting.”
Backplay: "Man muss immer umkehren" (2018, unknown): “…that standard independent multi-agent learning does not produce agents that are able to cooperate in social dilemmas (Leibo et al., 2017; Lerer & Peysakhovich, 2017; Peysakhovich & Lerer, 2017; Foerster et al., 2017) or risky coordination games (Yoshida et al., 2008; Peysakhovich & Lerer, 2018).”
Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning (2020, background): “Most mixed-motive reinforcement learning research has produced policies through self-play [32, 43] or co-training policies in fixed groups [24, 31].”