Social reward shaping in the prisoner's dilemma

By Monica Babes-Vroman, Enrique Munoz de Cote, Michael L. Littman. 2008

72 citations recorded by the source corpus.

Reward shaping is a well-known technique applied to help reinforcement-learning agents converge more quickly to near-optimal behavior. In this paper, we introduce social reward shaping, which is reward shaping applied in the multiagent-learning framework. We present preliminary experiments in the iterated Prisoner's dilemma setting that show that agents using social reward shaping appropriately can behave more effectively than other classical learning and non-learning strategies. In particular, we show that these agents can both lead---encourage adaptive opponents to stably cooperate---and follow---adopt a best-response strategy when paired with a fixed opponent---where better known approaches achieve only one of these objectives.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “Other approaches can be found in Devlin et al. (2014); HolmesParker et al. (2016); Babes et al. (2008).”
Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “In RL, reward shaping is popular for learning desired behavior in various domains (Ng et al., 1999; Babes et al., 2008; Devlin & Kudenko, 2011), which inspires our idea for ﬁnding diverse strategic behavior.”
"Other-Play" for Zero-Shot Coordination (2020, background): “Here the main problem is that SP may reach inefﬁcient equilibria and so there is a large literature on pushing self-play toward higher payoff equilibria using various algorithmic innovations (Babes et al., 2008; Devlin et al., 2011; Devlin & Kudenko, 2016; Peysakhovich & Lerer, 2018).”
Maintaining cooperation in complex social dilemmas using deep reinforcement learning (2017, background): “A third literature focuses on situations where long term interactions with the same partner means that a good agent needs to either to discern a partner’s type (Littman, 2001) or be able shape the adaptation of a learning partner (Babes et al., 2008; Foerster et al., 2017b).”
Theoretical considerations of potential-based reward shaping for multi-agent systems (2011, methodology): “The remaining applications that were potential-based [2, 16], demonstrated an increased probability of convergence to a higher value Nash Equilibrium.”
Autonomous Agents and Multi-agent Systems Manuscript No. Learning Potential Functions and Their Representations for Multi-task Reinforcement Learning (unknown): “In practice, it has also been shown to improve strategies for the iterated Prisoner’s dilemma [4] and robot soccer [10].”
Prosocial learning agents solve generalized Stag Hunts better than selfish ones (2017, background): “If we know the structure of the game, we can choose an appropriate potential function (Babes et al., 2008; Devlin and Kudenko, 2011; Devlin et al., 2011).”
Emotional Multiagent Reinforcement Learning in Spatial Social Dilemmas (2015, background): “For example, Sandholm and Crites [20] and Masuda and Ohtsuki [21] defined local state as the sensations of past actions of the interacting agents, while [22]–[24] evolved or reshaped individual rewards during interactions so that agents could rapidly achieve cooperation.”
Consequentialist conditional cooperation in social dilemmas with imperfect information (2017, background): “A related set of work looks at learning can be shaped to converge to better outcomes (Babes et al., 2008).”
An exploration strategy for non-stationary opponents (2017, background): “In this cases, a hard-reset like we propose may not be the best approach and we leave that as an open question where ideas from safe-exploration [22,24] or social reward shaping [3] might be worth exploring.”
Lenient Learning in Independent-Learner Stochastic Cooperative Games (2016, unknown): “Asymmetric Q-learning (Könönen, 2004), distinguishes between “leaders” and ”followers” in the game, where a “leader” knows what action a “follower” will choose can guide the action selection of the “follower” (see also the leader-follower reward shaping in Babes et al. 2008).”