Potential-based difference rewards for multiagent reinforcement learning

By Sam Devlin, Logan Michael Yliniemi, Daniel Kudenko, Kagan Tumer. 2014

118 citations recorded by the source corpus.

Difference rewards and potential-based reward shaping can both significantly improve the joint policy learnt by multiple reinforcement learning agents acting simultaneously in the same environment. Difference rewards capture an agent's contribution to the system's performance. Potential-based reward shaping has been proven to not alter the Nash equilibria of the system but requires domain-specific knowledge. This paper introduces two novel reward functions that combine these methods to leverage the benefits of both. Using the difference reward's Counterfactual as Potential (CaP) allows the application of potential-based reward shaping to a wide range of multiagent systems without the need for domain specific knowledge whilst still maintaining the theoretical guarantee of consistent Nash equilibria. Alternatively, Difference Rewards incorporating Potential-Based Reward Shaping (DRiP) uses potential-based reward shaping to further shape difference rewards. By exploiting prior knowledge of a problem domain, this paper demonstrates agents using this approach can converge either up to 23.8 times faster than or to joint policies up to 196% better than agents using difference rewards alone.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, background): “However, even in the single-agent case, reward shaping is difﬁcult and only a small class of shaped reward functions are guaranteed to preserve optimality w.r.t. the true objective (Ng et al., 1999; Devlin et al., 2014; Eck et al., 2016).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “A solution that came fromMAL research that has proven successful in many scenarios is difference rewards [3,86,332], which aims to capture an agent’s contribution to the system’s global performance.”
Multi-objective multi-agent decision making: a utility-based analysis and survey (2019, background): “While it is possible for multiple individual learners in a cooperative MAS to converge to a point of equilibrium, whether they will converge to an optimal joint policy (one which maximises the system welfare) depends on the specific learning algorithm and reward scheme used [42,89,122].”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, methodology): “RS has also been applied to multi-agent RL [82] and model-based RL [83].”
Exploiting Structure and Agent-Centric Rewards to Promote Coordination in Large Multiagent Systems (2014, background): “Difference rewards are a type of shaping reward in the form [6, 9, 16, 18, 17, 40]:”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “In order to address this issue, [31] combines these approaches and proposes two tabular algorithms.”
Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning (2018, background): “A similar paper shows that counterfactuals can be used with potential-based reward shaping to improve credit assignment for training a joint policy in multi-agent RL (Devlin et al., 2014).”
Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving (2019, methodology): “or mixed cooperative and competitive environments; MADDPG is a similar idea as that in Foerster et al. [2018]. Other related work Our proposed AdvRA and CI are varieties of credit assignment methods [Devlin et al., 2014; Nguyen et al., 2018]. Most existing methods allocate the proper reward to each ”
Exploiting structure and utilizing agent-centric rewards to promote coordination in large multiagent systems (2013, background): “Difference rewards are a type of shaping reward in the form [6, 9, 16, 18, 17, 40]:”
On Multi-Agent Learning in Team Sports Games (2019, methodology): “We also plan to better understand and explore multi-agent credit assignment in this environment (Devlin et al., 2014).”
A Theoretical and Empirical Analysis of Reward Transformations in Multi-Objective Stochastic Games (2017, result): “It extends an earlier single-objective version [7], in a similar manner to Yliniemi and Tumer’s multi-objective extension [21] to the El-Farol bar problem [1].”