An Algorithm for Distributed Reinforcement Learning in Cooperative Multi-Agent Systems

By Martin Lauer, Martin A. Riedmiller. 2000

531 citations recorded by the source corpus.

Subfields: Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In ML Systems & Hardware

Citations in the corpus

Massively Parallel Methods for Deep Reinforcement Learning (2015, background): “A signiﬁcant part of this work has focused on distributed multi-agent systems (Weiss, 1995; Lauer & Riedmiller, 2000).”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Lauer and Riedmiller (2000) ignore updates which decrease Q -value estimates in order to not prematurely underestimate an action’s Q -value due to the exploratory actions of other agents.”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “Most studied are cooperative settings, with strategies such as optimistic and hysteretic Q function updates [15, 21, 25], which assume that the actions of other agents are made to improve collective reward.”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, background): “was introduced in partially observable settings where messages passing between agents areallowed.Theagentisalsonamedasa independentlearner . The idea of learning independent agents can also be found (Lauer and Riedmiller 2000;KapetanakisandKudenko2002; Lauer and Riedmiller 2004;Foerster et al .2016)”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “Target network parameters ˆ θ j are updated less frequently and, in combination with experience replay, are critical for stable deep Q-learning.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, background): “Lauer & Riedmiller (2000) demonstrate a variation of the coordination-free method called distributed Q-learning.”
Learning Multiagent Communication with Backpropagation (2016, methodology): “A ongst fully cooperative algorithm , many approaches [12, 15, 33] av id the need for communicat on by m king tr ng assumptions bout v sibility f oth r agents and the vironment.”
QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, background): “nment induced by simultaneously learning and exploring agents. Foerster et al. (2017) addresses the issue of non-stationarity when using an experience replay with independent learners to some extent. Lauer and Riedmiller (2000) ignore updates which decrease Q-value estimates in order to not prematur”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “Another avenue for improvement is to extend DDRQN to make use of various multi-agent adaptations of Q-learning (Tan, 1993; Littman, 1994; Lauer & Riedmiller, 2000; Panait & Luke, 2005).”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, methodology): “Q-Learning (Watkins & Dayan, 1992) is a popular model-free method, which iterates on a set of Q values or approx-imators to approach optimal Q values or estimates, where Q ( s, a ) is the expected maximum sum of rewards achievable in the future given state s and action a .”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, methodology): “…state while s is not an absorbing state do From s select a according to the ǫ -greedy selection method based on π i Apply a and observe reward r and next state Lauer and Riedmiller (2000) proposed the distributed Q-learning (algorithm 1) that addresses three factors that can cause mis-coordination.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “, distributed Q-learning [225] and hysteretic Q-learning [8]) lenient learners initially maintain an optimistic disposition to mitigate the noise from transitions resulting in miscoordination, preventing agents from being drawn towards sub-optimal but wide peaks in the reward search space [97].”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “In addition, for MARL, cooperation and competition between agents has been studied [51,70,60,12,69,36,58,28,57].”
Making friends on the fly: Cooperating with new teammates (2017, unknown): “An example of this approach is Lauer and Riedmiller’s work which introduces Distributed Q-learning [53] for learning in multiagent, fully cooperative settings.”
Coordination in multiagent reinforcement learning: a Bayesian approach (2003, methodology): “Lauer and Riedmiller [11] describe a Q-learning method for identical interest stochastic games that explicitly embodies this optimistic assumption in its Q-value estimates.”
QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning (2020, methodology): “Indeed, naïvely applying single-agent algorithms demonstrated underwhelming results (Tan, 1993; Lauer and Riedmiller, 2000; Tampuu et al., 2017).”
Predicting and Preventing Coordination Problems in Cooperative Q-learning Systems (2007, methodology): “[Claus and Boutilier, 1998; Lauer and Riedmiller, 2000; Littman, 2001; Wang and Sandholm, 2002].”
Failure-Scenario Maker for Rule-Based Agent using Multi-agent Adversarial Reinforcement Learning and its Application to Autonomous Driving (2019, unknown): “Most previous work on MARL addresses cooperative tasks, in which the cumulative reward is maximized as a group [Lauer and Riedmiller, 2000; Panait and Luke, 2005].”
Multi-agent deep reinforcement learning: a survey (2021, background): “The former ignores the existence of other agents and cannot observe the rewards and selected actions of others as considered in Bowling and Veloso (2002) and Lauer and Riedmiller (2000).”
QOPT: Optimistic Value Function Decentralization for Cooperative Multi-Agent Reinforcement Learning (2020, result): “This is similar to an optimistic agent proposed by Lauer and Riedmiller [9] that ignores the low target action-value when learning the optimal action with independent action-value estimators, which have representational limitations in a fully distributed setting.”