Lenient Multi-Agent Deep Reinforcement Learning

By Gregory Palmer, Karl Tuyls, Daan Bloembergen, Rahul Savani. 2017

168 citations recorded by the source corpus.

Much of the success of single agent deep reinforcement learning (DRL) in recent years can be attributed to the use of experience replay memories (ERM), which allow Deep Q-Networks (DQNs) to be trained efficiently through sampling stored state transitions. However, care is required when using ERMs for multi-agent deep reinforcement learning (MA-DRL), as stored transitions can become outdated when agents update their policies in parallel \citefoerster2017stabilising. In this work we apply leniency \citepanait2006lenient to MA-DRL. Lenient agents map state-action pairs to decaying temperature values that control the amount of leniency applied towards negative policy updates that are sampled from the ERM. This introduces optimism in the value-function update, and has been shown to facilitate cooperation in tabular fully-cooperative multi-agent reinforcement learning problems. We evaluate our Lenient-DQN (LDQN) empirically against the related Hysteretic-DQN (HDQN) algorithm \citeomidshafiei2017deep as well as a modified version we call scheduled -HDQN, that uses average reward learning near terminal states. Evaluations take place in extended variations of the Coordinated Multi-Agent Object Transportation Problem (CMOTP) \citebucsoniu2010multi. We find that LDQN agents are more likely to converge to the optimal policy in a stochastic reward CMOTP compared to standard and scheduled-HDQN agents.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Wei and Luke (2016) show that Leniency outperforms Hysteretic Q -learning in cooperative stochastic games, and Palmer et al. (2018) extend Leniency to the deep RL setting and show beneﬁts over Hysteretic and fully Independent Q -learners on deterministic and stochastic gridworlds with two agents.”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, background): “More recently, Palmer et al. (2018) extended the idea of leniency (Potter & Jong, 1994; Panait et al., 2008) to deep MARL and proposed the retroactive temperature decay schedule to address stochastic rewards problems.”
The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning (2019, background): “[23] apply leniency to independent double DQN learners in a coordinated grid world problem, while Foerster et al.”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “To date the majority of MA-DRL research focuses on stochasticity and mitigating an amplified moving target problem resulting from obsolete state transitions being stored inside Experience Replay Memories (ERM) [4, 18, 19, 30].”
Deep Reinforcement Learning for Swarm Systems (2018, methodology): “Following this work, Palmer et al. [2017] add leniency [Panait et al., 2006] to the hysteretic approach to prevent “relative overgener-alization” of agents.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “parameter sharing: PS-DQN, PS-DDPG and PS-TRPO, which extended single-agent DQN, DDPG and TRPO algorithms, respectively. The results showed that PS-TRPO outperformed the other two. Lenient-DQN (LDQN) [35] took the leniency concept [219] (originally presented in MAL) and extended their use to MDRL. T”
Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, background): “…al. (2017); Foerster et al. (2017a); Omidshaﬁei et al. (2017); Foerster et al. (2016); Sukhbaatar et al. (2016); Mordatch & Abbeel (2017); Havrylov & Titov (2017); Palmer et al. (2017); Peng et al. (2017); Fo-erster et al. (2017c); Tampuu et al. (2017); Leibo et al. (2017); Foerster et al. (2017b).”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, methodology): “Lenient Deep Q-Network (LDQN) [28] incorporates lenient learning [29] with DQN by encoding the high-dimensional state space into lower dimensions where temperature values are feasible to be stored and updated.”
Neural Replicator Dynamics (2019, background): “Most other works that focus on nonstationary multiagent learning are limited to repeated and stochastic games [1, 15, 24, 46, 47].”
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review (2020, background): “To this end, several works try to add information to the experience tuple to help the algorithm adapt to MADRL settings [35]-[37].”
Multi-Agent Reinforcement Learning: A Review of Challenges and Applications (2021, methodology): “The agents share the parameters of their critic to reach consensus 3050 Lenient deep Qnetwork [32] Value based Independent agents Stores a temperature variable in the experience replay to decide the amount of leniency to apply to the updates 97”
Learning to Interactively Learn and Assist (2019, background): “When a broader range of interactive behaviors is desired, prior works have introduced a multi-agent learning component (Potter and Jong 1994; Palmer et al. 2018).”
Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning (2020, background): “Cooperative multiagent reinforcement learning problem has been studied extensively in the last decade, where a system of agents learn towards coordinated policies to optimize the accumulated global rewards (Busoniu et al., 2008; Gupta et al., 2017; Palmer et al., 2018).”
Q-value Path Decomposition for Deep Multiagent Reinforcement Learning (2020, background): “Cooperative multiagent reinforcement learning problem has been studied extensively in the last decade (Busoniu et al., 2008; Gupta et al., 2017; Palmer et al., 2018), where a system of agents learn towards coordinated policies to optimize 1Tianjin Univeristy 2Tencent 3NetEase Fuxi AI Lab 4Fudan…”
Multi-agent deep reinforcement learning: a survey (2021, background): “Another extension for experience replays was proposed by Palmer et al. (2018) who applied leniency to every stored transition sample.”
Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments (2018, methodology): “As for the non-stationary coexisting agents, we incorporate the leniency from lenient Q-learning [ Palmer et al. , 2018; Panait et al. , 2006] into the reward approximator to provide an optimistic estimation of the expected reward under each state-action pair r ( s, a ) .”