Factorized Q-learning for large-scale multi-agent systems

By Yong Chen, M. Zhou, Ying Wen, Yaodong Yang, Yufeng Su, Weinan Zhang, Dell Zhang, Jun Wang, Han Liu. 2018

75 citations recorded by the source corpus.

Deep Q-learning has achieved significant success in single-agent decision making tasks. However, it is challenging to extend Q-learning to large-scale multi-agent scenarios, due to the explosion of action space resulting from the complex dynamics between the environment and the agents. In this paper, we propose to make the computation of multi-agent Q-learning tractable by treating the Q-function (w.r.t. state and joint-action) as a high-order high-dimensional tensor and then approximate it with factorized pairwise interactions. Furthermore, we utilize a composite deep neural network architecture for computing the factorized Q-function, share the model parameters among all the agents within the same group, and estimate the agents' optimal joint actions through a coordinate descent type algorithm. All these simplifications greatly reduce the model complexity and accelerate the learning process. Extensive experiments on two different multi-agent problems demonstrate the performance gain of our proposed approach in comparison with strong baselines, particularly when there are a large number of agents.

Subfields: Artificial Intelligence, Computational Mathematics, Electrical and Electronic Engineering, Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

In ML Systems & Hardware

Citations in the corpus

Deep Coordination Graphs (2019, methodology): “id et al., 2018). We extend this approach with payoff functions speciﬁed by pairwise edges in a given CG (Guestrin et al., 2002a). We take inspiration from highly scalable methods (Yang et al., 2018; Chen et al., 2018) and improve over SCQL (Kok & Vlassis, 2006) and the approach of Castellini et”
From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, background): “Moreover, in large-scale multiagent systems, the dynamics and stochasticity of the environment become more complex, which makes it more challenging to achieve coordination among agents (Chen et al. 2018).”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “Chen et al. (2018) decomposed the joint action-value function into independent components and used pairwise interactions between agents to render large-scale problems computationally tractable.”
Multi-Agent Game Abstraction via Graph Attention Neural Network (2019, background): “Recent work has focused on multi-agent reinforcement learning in large-scale multi-agent systems (Yang et al. 2018; Chen et al. 2018), in which the large number of”
Towards Open Ad Hoc Teamwork Using Graph-based Policy Learning (2020, methodology): “(7) Previous work from Zhou et al. (2019) demonstrated that low-rank factorization enables scalable pairwise utility computation even under thousands of possible pairwise actions.”