The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems

By Caroline Claus, Craig Boutilier. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, neutral): “Centralized methods [6] directly learn a single policy to produce the joint actions of all agents.”
Value-Decomposition Networks For Cooperative Multi-Agent Learning (2017, methodology): “Because of this inability to explain its own observed rewards naive independent RL is often unsuccessful: for example Claus and Boutilier (1998) show that independent Q -learners cannot distinguish teammates’ exploration from stochasticity in the environment, and fail to solve even an apparently…”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, background): “nd shows its potential usage in a wide range of multiagent tasks in the real-world applications. 2 Related Work The studies on interaction and collaboration in multiagent settings have a long history [24, 25, 18, 26]. Although limited to toy examples in the beginning, reinforcement learning, as a me”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “Claus & Boutilier (1998) dichotomize MARL agents into two classes: Joint Action Learners (JALs) and Independent Learners (ILs).”
Multiagent cooperation and competition with deep reinforcement learning (2015, background): “This is an extension to the existing multiagent reinforcement learning studies done in simple grid worlds or with agents already equipped with abstract high-level sensory perception [3, 12, 13].”
Empirically Evaluating Multiagent Learning Algorithms (2014, methodology): “While four of the eleven algorithms tested in this study were simple stationary-strategy baselines, the remaining seven were MAL algorithms including Hyper-Q (Tesauro, 2004), WoLF-PHC (Bowling and Veloso, 2002), and a joint action learner (Claus and Boutilier, 1997).”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “IPPO with policy and value clipping [6] IPPO without value clipping [5] IPPO without policy clipping [5] IPPO without clipping [1] IPPO without clipping, with low learning rate [4] (c) 6h vs 8z (d) MMM2 IPPO with policy and value clipping [5] IPPO without value clipping [5] IPPO without policy…”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Joint action learning (Claus & Boutilier, 1998) learns centralized policies conditioned on the full state, but this global information is often unavailable during execution due to partial observability or communication constraints.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “Tampuu et al. [30] train each agent’s DQN [19] using Independent Q -learning [31], which treats the other agents as part of the environment, which can lead to many pathologies and instability during training [4, 8].”
Learning with Opponent-Learning Awareness (2017, methodology): “Detailed studies have analysed the dynamics of JALs in general sum settings: This includes work by Uther and Veloso [43] in zero-sum settings and by Claus and Boutilier [8] in cooperative settings.”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, background): “(Claus and Boutilier 1998; Busoniu, Babuska, and Schutter 2008).”
Action Semantics Network: Considering the Effects of Actions in Multiagent Systems (2019, background): “However, there also exist a lot of challenges in multiagent systems (MASs) since agents’ behaviors are influenced by each other and the environment exhibits more stochasticity and uncertainties (Claus & Boutilier, 1998; Hu & Wellman, 1998; Bu et al., 2008; Hauwere et al., 2016).”
The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning (2019, background): “action values Qi (si ,ai ) [15, 25], that are known to be hard to learn because of non-stationarities from the perspective of a single agent due to the simultaneous learning of the others [4, 25, 28].”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, background): “This decentralization can be more scalable, but poses issues not associated with centralized or joint learning where actions are shared across agents (Claus & Boutilier, 1998).”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, methodology): “Thus, team games are fully cooperative settings, where players have a shared objective to maximize their common return (see Figure 1) [2, 3, 16].”
Factorized Q-learning for large-scale multi-agent systems (2018, background): “In the best-response learning paradigm, each agent just tries to learn a policy that is optimal with respect to the joint policy of the other players [2, 26].”
Multiagent Learning: Basics, Challenges, and Prospects (2012, methodology): “Therefore extensions of the MDP framework have been considered such as Markov games and joint action learners (Littman 1994, Claus and Boutilier 1998).”
Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems (2012, methodology): “We focus on non-communicative agents called independent learners (ILs) that are unable to observe the actions of the other agents (Claus & Boutilier, 1998).”