A multi-agent reinforcement learning model of common-pool resource appropriation

By Julien Pérolat, Joel Z. Leibo, Vinícius Flores Zambaldi, Charlie Beattie, Karl Tuyls, Thore Graepel. 2017

225 citations recorded by the source corpus.

Humanity faces numerous problems of common-pool resource appropriation. This class of multi-agent social dilemma includes the problems of ensuring sustainable use of fresh water, common fisheries, grazing pastures, and irrigation systems. Abstract models of common-pool resource appropriation based on non-cooperative game theory predict that self-interested agents will generally fail to find socially positive equilibria---a phenomenon called the tragedy of the commons. However, in reality, human societies are sometimes able to discover and implement stable cooperative solutions. Decades of behavioral game theory research have sought to uncover aspects of human behavior that make this possible. Most of that work was based on laboratory experiments where participants only make a single choice: how much to appropriate. Recognizing the importance of spatial and temporal resource dynamics, a recent trend has been toward experiments in more complex real-time video game-like environments. However, standard methods of non-cooperative game theory can no longer be used to generate predictions for this case. Here we show that deep reinforcement learning can be used instead. To that end, we study the emergent behavior of groups of independently learning agents in a partially observed Markov game modeling common-pool resource appropriation. Our experiments highlight the importance of trial-and-error learning in common-pool resource appropriation and shed light on the relationship between exclusion, sustainability, and inequality.

Subfields: Management Science and Operations Research, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Safety Research

In Sociology and Political Science

Citations in the corpus

Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Many stages of emergent strategy can be mapped to behavioral shifts in the way agents interact with the tools in their environment, similar to [61, 62].”
Learning Attentional Communication for Multi-Agent Cooperation (2018, background): “COMA [4] is proposed to solve multi-agent credit assignment in cooperative settings, and resource appropriation in MARL is studied in [19].”
Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning (2018, methodology): “In recent years, a variety of deep RL methods have been adapted to this setting (Foerster et al., 2016a; Lowe et al., 2017; Perolat et al., 2017; Jaderberg et al., 2018).”
Relational Forward Models for Multi-Agent Learning (2018, positive): “There is thus an increasing focus on multi-agent systems that learn how to coordinate on their own (Jaderberg et al., 2018; Pachocki et al., 2018; Perolat et al., 2017).”
Consequentialist conditional cooperation in social dilemmas with imperfect information (2017, background): “) affect the dynamics of behavior (Fudenberg & Levine, 1998; Sandholm & Crites, 1996; Shoham et al., 2007; Nowak, 2006; Conitzer & Sandholm, 2007; Leibo et al., 2017; Perolat et al., 2017).”
Maintaining cooperation in complex social dilemmas using deep reinforcement learning (2017, background): “Leibo et al. (2017); Perolat et al. (2017) study standard RL agents in more complex social dilemmas but are primarily concerned with describing how incentives affect the emergence of cooperative or non-cooperative behavior.”
Inequity aversion improves cooperation in intertemporal social dilemmas (2018, background): “Explicitly, a Schelling diagram [45, 18] depicts the relative payoffs for a single cooperator or defector given a fixed number of other cooperators.”
Multi-Agent Reinforcement Learning: A Review of Challenges and Applications (2021, methodology): “MARL was used in social sciences in common pool resource (CPR) appropriation [66] and the study of sequential social dilemmas (SSD) [67].”
Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research (2019, background): “(Hughes et al., 2018; Perolat et al., 2017) extended its approach beyond the two-player case to analyze the strategic incentives underlying policies learned by reinforcement learning in common pool resource appropriation and public goods scenarios.”
Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning (2018, background): “A Schelling diagram (Schelling, 1973; Perolat et al., 2017) depicts the relative payoffs for a single agent’s strategy given a fixed number of other agents who are cooperative.”
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, methodology): “lized to characterize self-interested independent learning agents to nd equilibria of the SSD, which cannot be solved by the standard evolution and learning methods used for MGSD [43]. Perolat et al. [79] also demonstrated 16 the application of MADRL in social science phenomenon, i.e. the common-poo”
Multi-agent deep reinforcement learning: a survey (2021, background): “Nevertheless, if the contemporary dynamics of the learners are neglected, algorithms can utilize short-term buffers as applied in Baker et al. (2020) and Leibo et al. (2017).”
Prosocial learning agents solve generalized Stag Hunts better than selfish ones (2017, background): “…two game types have very different properties from the point of view of learning/evolutionary dynamics, how humans act in them and in how one would construct agents to solve them in the real world (Lerer and Peysakhovich 2017; Leibo et al. 2017; Kleiman-Weiner et al. 2016; Perolat et al. 2017).”
Social Diversity and Social Preferences in Mixed-Motive Reinforcement Learning (2020, result): “These resemble previous approaches using selfishness [42], inequity aversion [24, 56], and strong reward sharing [44, 52], respectively.”