Multi Agent Reinforcement Learning Independent vs Cooperative Agents

By Ming Tan. 2003

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “Perhaps the most commonly applied method in multi-agent learning is independent Q-learning (IQL) (Tan, 1993), which decomposes a multi-agent problem into a collection of simultaneous single-agent problems that share the same environment.”
Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication (2020, neutral): “Independent Q-Learning (IQL) : Similar to DQN which learns an action-value approximator Q π ( s, a ; θ ), IQL learns to approximate Q π n ( o n , a n ; θ n ) with respect to each agent in the multi-agent setting [20].”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “The pre-trained pool consists of agents trained with different MARL methods such as Independent Q-learning (IQL) (Tan, 1993), Value Decomposition Networks (VDN) (Sunehag et al., 2017), Simpliﬁed Action Decoder (SAD) (Hu & Foerster, 2019), Other-Play (OP) (Hu et al., 2020) with different…”
Learning Multi-Arm Manipulation Through Collaborative Teleoperation (2020, background): “Multi-Agent Reinforcement Learning: Multi-Agent Re-inforcement Learning [5, 35] in cooperative settings has been widely studied [11, 12, 14, 21, 26, 34], and applied to domains such as video games [28] and visual question answering [9].”
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “QN is the use of an experience replay buffer Dcontaining tuples (s;a;r;s0). Q-Learning can be directly applied to multi-agent settings by having each agent ilearn an independently optimal function Q i[36]. However, because agents are independently updating their policies as learning progresses, the ”
Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games (2017, background): “nd shows its potential usage in a wide range of multiagent tasks in the real-world applications. 2 Related Work The studies on interaction and collaboration in multiagent settings have a long history [24, 25, 18, 26]. Although limited to toy examples in the beginning, reinforcement learning, as a me”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “They also evaluate a model combining Decentralized Q-learning with DRQNs, which they call Reinforced Inter-Agent Learning.”
Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, methodology): “The practical difﬁculty of hyper Q-learning is, of course, that the dimensionality of the Q-function has increased, making it potentially infeasible to learn.”
Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks (2016, background): “Some work has looked at learning control policies for individual agents in a collaborative setting with communication constraints [38, 3], with applications such as soccer robot control [31], and methods such as hierarchical reinforcement learning for communicating high-level goals [12], or…”
Learning Multiagent Communication with Backpropagation (2016, background): “In all modules, the hidden layer size is set to 50.”
Multiagent cooperation and competition with deep reinforcement learning (2015, background): “This is an extension to the existing multiagent reinforcement learning studies done in simple grid worlds or with agents already equipped with abstract high-level sensory perception [3, 12, 13].”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, background): “Knowledge sharing in MARL From IQL [34] to QPLEX, many works focus on designing mixing network structures and have provided promising empirical and theoretical results.”
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing (2021, background): “Tan (1993) investigates sharing policies between cooperative settings in non-deep RL settings.”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, background): “While easy to distribute and decentralisable by construction, IL suffers from a variety of theoretical limitations that may result in learning instabilities or suboptimal performance observed in practice [27, 9, IQL, IAC].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Independent learning (Tan, 1993) avoids this problem by learning decentralized policies but suffers from non-stationarity during learning as it treats other learning agents as part of the environment.”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “We compare our method with five state-of-the-art baselines: QTRAN [8], Qatten [9], QMIX [7], VDN [6], and independent Q-learning (IQL) [21] in the didactic and StarCraft II tasks.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “[30] train each agent’s DQN [19] using Independent Q-learning [31], which treats the other agents as part of the environment, which can lead to many pathologies and instability during training [4, 8].”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, methodology): “In RWARE, VDN and QMIX do not exhibit any learning, similar to IQL, COMA and MADDPG, indicating that value decomposition methods require sufficiently dense rewards to successfully learn to decompose the value function into the individual agents.”