QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning

By Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, Yung Yi. 2019

1,068 citations recorded by the source corpus.

We explore value-based solutions for multi-agent reinforcement learning (MARL) tasks in the centralized training with decentralized execution (CTDE) regime popularized recently. However, VDN and QMIX are representative examples that use the idea of factorization of the joint action-value function into individual ones for decentralized execution. VDN and QMIX address only a fraction of factorizable MARL tasks due to their structural constraint in factorization such as additivity and monotonicity. In this paper, we propose a new factorization method for MARL, QTRAN, which is free from such structural constraints and takes on a new approach to transforming the original joint action-value function into an easily factorizable one, with the same optimal actions. QTRAN guarantees more general factorization than VDN or QMIX, thus covering a much wider class of MARL tasks than does previous methods. Our experiments for the tasks of multi-domain Gaussian-squeeze and modified predator-prey demonstrate QTRAN's superior performance with especially larger margins in games whose payoffs penalize non-cooperative behavior more aggressively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, methodology): “Value-decomposition (VD) methods are another class of CTDE algorithms which represent the joint Q-function as a function of agents’ local Q-functions [32, 27, 31] and have established state of the art results in popular MARL benchmarks [37, 36].”
Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “One promising approach to implement the CTDE framework is value function factorization.”
Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, background): “A variety of recent approaches employ joint optimisation of, or transfer learning between, factored and unrestricted joint value functions in order to overcome the representational limitations of factored value functions, with varying success [4, 24].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “They implicitly represent the coordination dependencies among agents by the decomposable structure (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019; Wang et al., 2020e;b).”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, methodology): “QTRAN (Son et al., 2019) constructs two soft regularizations to align the greedy action selections between the joint and individual value functions.”
Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, background): “QTRAN [27] is another Q -learning based algorithm that learns an unrestricted joint action Q function and aims to solve a constrained optimisation problem in order to decentralise it.”
The StarCraft Multi-Agent Challenge (2019, methodology): “PyMARL features implementations of the following algorithms: QMIX (Rashid et al., 2018), QTRAN (Son et al., 2019) and COMA (Foerster et al., 2018a) as state-of-the-art methods, and VDN (Sunehag et al., 2017) and IQL (Tan, 1993) as baselines.”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, methodology): “choose from a diverse set of cooperative MARL algorithms with decentralized execution [10, 32, 36, 44].”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “An important concept for such methods is decentralisability (see IGM in [44]) which asserts that ∃qi, such that ∀s,u:”
Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, background): “Three representative examples: VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) gradually increase the representation ability of the mixing network.”
Deep Coordination Graphs (2019, methodology): “one agent are thus used to train all. This can dramatically improve the sample efﬁciency compared to unfactored methods (Foerster et al., 2016; 2018; Lowe et al., 2017; Schroder de Witt et al., 2019; Son et al., 2019), which correspond to¨ a CG with one hyper-edge connecting all agents (Figure 1c). ”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, methodology): “VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QTRAN (Son et al., 2019) have progressively enlarged the family of functions that can be represented by the mixing network.”
Neighborhood Cognition Consistent Multi-Agent Reinforcement Learning (2019, background): “…centralized critic (Lowe et al. 2017; Foerster et al. 2018; Mao et al. 2019), joint value factorization (Sunehag et al. 2018; Rashid et al. 2018; Son et al. 2019) and agent communication (Fo-erster et al. 2016; Sukhbaatar, Fergus, and others 2016; Peng et al. 2017), the neighborhood cognitive…”
Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (2022, background): “This value-functional representation of agents’ actions enables studying interactions between them, as well to decompose the joint value function signal, thus helping alleviate the severity of the credit assignment problem [29, 35, 22].”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, methodology): “Beneﬁting from a richer value function class, it is shown that QTRAN [14] achieves a little more stable performance while training with a static dataset.”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, methodology): “We adopt the centralized training and decentralized execution (CTDE) paradigm [43].”