Multiagent Soft Q-Learning

By Ermo Wei, Drew Wicke, David Freelan, Sean Luke. 2018

87 citations recorded by the source corpus.

Policy gradient methods are often applied to reinforcement learning in continuous multiagent games. These methods perform local search in the joint-action space, and as we show, they are susceptable to a game-theoretic pathology known as relative overgeneralization. To resolve this issue, we propose Multiagent Soft Q-learning, which can be seen as the analogue of applying Q-learning to continuous controls. We compare our method to MADDPG, a state-of-the-art approach, and show that our method achieves better coordination in multiagent cooperative tasks, converging to better local optima in the joint action space.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “Algorithms which suffer from relative overgeneralisation [23, 33], or which make poor trade-offs in their representation (as VDN and QMIX do) can fail to solve this task.”
QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, background): “Wei et al. (2018) proposes MultiAgent Soft Q-learning in continuous action spaces to tackle the relative overgeneralization problem (Wei & Luke, 2016) and achieves better coordination.”
Deep Coordination Graphs (2019, methodology): “Wei et al. (2018) 2018) introduce a value factorization for a similar setup based on a low-rank approximation of the joint value.”
Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces (2019, methodology): “In this work, we consider a Cooperative Stochastic Game [Wei et al. , 2018a] in partially observable settings modeled as a tuple { S, U, r, P, γ, H, N } .”
Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, background): “(2017), where the other agents’ actions are sampled from the replay buffer, potentially causing overgeneralization where agents fail to coordinate based on their current policies (Wei et al., 2018).”
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning (2018, background): “In contrast to our approach Multiagent Soft Q-Learning augments rewards with an entropy term, thereby increasing the likelihood of finding multiple modes within a continuous action space compared to the discrete action space used in our experiments [8, 29].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “nt scenarios. However, learning in multiagent environments is fundamentally more dicult due to non-stationarity, the increase of dimensionality, and the creditassignment problem, among other factors [1, 5, 10, 49, 90, 126]. While previous MAL surveys have tried to contextualize most of the MAL liter”
Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning (2019, unknown): “…with decentralized execution” in the deep RL domain, including MADDPG (Lowe et al., 2017), COMA (Foerster et al., 2017), MF-AC (Yang et al., 2018), Multi-Agent Soft-Q (Wei et al., 2018), and LOLA (Foerster et al., 2018), can be classiﬁed into this category (see more clariﬁcations in Appendix B).”
Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient (2019, methodology): “…policy gradient algorithm with centralized baseline; Lowe et al. (2017) extended DDPG to multi-agent setting with a centralized Q function; Wei et al. (2018) and Grau-Moya (2018) proposed multi-agent variants of the soft-Q-learning algo-rithm (Haarnoja et al. 2017); Yang et al. (2018)…”
Multi-agent deep reinforcement learning: a survey (2021, background): “Further works that incorporate one centralized critic for distributed policies can be found in Das et al. (2019), Iqbal and Sha (2019) and Wei et al. (2018).”
FOP: Factorizing Optimal Joint Policy of Maximum-Entropy Multi-Agent Reinforcement Learning (2021, methodology): “When facing tasks with continuous actions, multi-agent actor-critic methods (Lowe et al., 2017; Foerster et al., 2018; Wei et al., 2018; Iqbal & Sha, 2019) are often taken into consideration.”
A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning (2020, methodology): “Considering the joint update from φ 0 to φ 1 , for simplicity, we can write the gradients in the inner-loop (see Equation (3)) based on the multiagent stochastic policy gradient theorem (Wei et al., 2018): where ρ φ 0 denotes the stationary distribution under the joint policy with parameters φ 0 .”