Actor-Attention-Critic for Multi-Agent Reinforcement Learning

By Shariq Iqbal, Fei Sha. 2018

970 citations recorded by the source corpus.

Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism which selects relevant information for each agent at every timestep. This attention mechanism enables more effective and scalable learning in complex multi-agent environments, when compared to recent approaches. Our approach is applicable not only to cooperative settings with shared rewards, but also individualized reward settings, including adversarial settings, as well as settings that do not provide global states, and it makes no assumptions about the action spaces of the agents. As such, it is flexible enough to be applied to most multi-agent learning problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, background): “Iqbal and Sha (2018) devise an actor-critic algorithm with per-agent critics that share an attention mechanism in order to improve scalability compared to a single centralised critic.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “PR2 (Wen et al., 2019) and MAAC (Iqbal & Sha, 2019) extend the CCDA paradigm by introducing the mechanism of recursive reasoning and attention, respectively.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “MADDPG is trained using the deterministic multi-agent policy-gradient theorem, via the Gumbel-Softmax trick, as in [15, 12].”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, neutral): “COMA (Foerster et al., 2018), MADDPG (Lowe et al., 2017), PR2 (Wen et al., 2019), and MAAC (Iqbal & Sha, 2019) explore multi-agent policy gradients.”
Learning Efficient Communication in Cooperative Multi-Agent Environment (2019, methodology): “Regarding our method and our experiments, we compare to four of the state-of-the-art approaches recently proposed for centralized training of decentralized policies: CommNet [13], COMA [4], MAAC [5], ATOC [7].”
A Review of Deep Reinforcement Learning for Smart Building Energy Management (2020, neutral): “DQN Value-based, off-policy, single-agent [28] Double DQN Value-based, off-policy, single-agent [72] Dueling DQN Value-based, off-policy, single-agent [73] Prioritized DQN Value-based, off-policy, single-agent [74] Distributional DQN Value-based, off-policy, single-agent [75] Noisy DQN Value-based, ”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms on a Building Energy Demand Coordination Task (2020, background): “However, this approach comes with other issues, for example scalability [6], given that the centralized critic’s joint state-action space can increase greatly with the addition of more buildings.”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, background): “Learning a centralised critic conditioning on a large joint agent observation space can be difﬁcult (Iqbal & Sha, 2019).”
Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning (2020, background): “Iqbal & Sha [25] introduced an attention-based critic to select agents in centralized training.”
Multi-agent deep reinforcement learning: a survey (2021, background): “Further works that incorporate one centralized critic for distributed policies can be found in Das et al. (2019), Iqbal and Sha (2019) and Wei et al. (2018).”
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, neutral): “Iqbal & Sha (2019) use an attention-based critic.”
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review (2020, positive): “Attention mechanism can be further integrated with this framework to enhance the perfor‐ mance of the MADRL algorithm [39].”
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, methodology): “…into two categories (Papoudakis et al., 2019): targeted modiﬁcations on standard RL learning schemes (Lowe et al., 2017; Foerster et al., 2018b; Iqbal & Sha, 2019; Baker et al., 2019), and opponent information estimation and sharing (Raileanu et al., 2018; Foerster et al., 2018a; Rabinowitz et…”
Q-value Path Decomposition for Deep Multiagent Reinforcement Learning (2020, methodology): “Using a modular network structure in the centralized critic is common in MARL algorithms and could effectively improve the performance (Iqbal & Sha, 2018; Liu et al., 2019).”
The Emergence of Individuality in Multi-Agent Reinforcement Learning (2020, methodology): “Some methods [15, 16, 13] extend policy gradient into multi-agent cases.”
Comparative Evaluation of Multi-Agent Deep Reinforcement Learning Algorithms (2020, background): “These algorithm classes are frequently used in the literature either as baselines or building blocks for more complex algorithms [8, 11, 13, 15, 17, 27, 29, 32].”