Inequity aversion improves cooperation in intertemporal social dilemmas

By Edward Hughes, Joel Z. Leibo, Matthew Phillips, Karl Tuyls, Edgar A. Duéñez-Guzmán, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin R. McKee, R. Koster and 2 more. 2018

271 citations recorded by the source corpus.

Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.

Subfields: Demography, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Sociology and Political Science

Citations in the corpus

RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “…the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be better explained from a MARL perspective.”
Relational Forward Models for Multi-Agent Learning (2018, positive): “…where multiple learning agents perceive and act in a shared environment, has produced impressive results (Jaderberg et al., 2018; Pachocki et al., 2018; Leibo et al., 2017; Hughes et al., 2018; Peysakhovich & Lerer, 2017a; Lerer & Peysakhovich, 2017; Bansal et al., 2017; Lanctot et al., 2017).”
Multi-agent deep reinforcement learning: a survey (2021, background): “Investigations on the trial-and-error learning in common-pool resource scenarios with multiple decision-makers are covered in Hughes et al. (2018), Pérolat et al. (2017) and Zhu and Kirley (2019).”
On the Utility of Learning about Humans for Human-AI Coordination (2019, background): “Some work tries to teach agents collaborative behaviors [22, 18] in environments where rewards are not shared across agents.”
Off-Policy Multi-Agent Decomposed Policy Gradients (2020, background): “Cooperative multi-agent reinforcement learning (MARL) has achieved great progress in recent years (Hughes et al., 2018; Jaques et al., 2019; Vinyals et al., 2019; Zhang et al., 2019; Baker et al., 2020; Wang et al., 2020c).”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “For example, social dilemmas will emerge from the relation of individual goals and the overall interests [118, 154].”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, background): “Since the intrinsic reward can be exactly assigned to the speciﬁc agent, individually maximizing the intrinsic reward is more efﬁcient than jointly maximizing the sum of all agents’ intrinsic rewards [29].”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “For example, social dilemmas will emerge from the relation between individual goals and overall interests [253], [254].”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, background): “Cooperative multi-agent reinforcement learning (MARL) has recently made great progress in many aspects by solving social dilemma (Hughes et al., 2018; Eccles et al., 2019), characterizing inﬂuence between agents (Jaques et al., 2019), optimizing both efﬁciency and fairness (Jiang & Lu, 2019),…”
Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research (2019, background): “For example. an institution for generating cooperation on the collective-level emerged in Hughes et al. (2018) when certain agents learned to sanction over-exploiters, effectively policing their behavior.”
Influence-Based Multi-Agent Exploration (2019, background): “Hughes et al . use inequality aversion reward to help intertemporal social dilemmas (Hughes et al., 2018).”