Multi-agent reinforcement learning as a rehearsal for decentralized planning
By Landon Kraemer, Bikramjit Banerjee. 2016
453 citations recorded by the source corpus.
Decentralized partially observable Markov decision processes (Dec-POMDPs) are a powerful tool for modeling multi-agent planning and decision-making under uncertainty. Prevalent Dec-POMDP solution techniques require centralized computation given full knowledge of the underlying model. Multi-agent reinforcement learning (MARL) based approaches have been recently proposed for distributed solution of Dec-POMDPs without full prior knowledge of the model, but these methods assume that conditions during learning and policy execution are identical. In some practical scenarios this may not be the case. We propose a novel MARL approach in which agents are allowed to rehearse with information that will not be available during policy execution. The key is for the agents to learn policies that do not explicitly rely on these rehearsal features. We also establish a weak convergence result for our algorithm, RLaR, demonstrating that RLaR converges in probability when certain conditions are met. We show experimentally that incorporating rehearsal features can enhance the learning rate compared to non-rehearsal-based learners, and demonstrate fast, (near) optimal performance on many existing benchmark Dec-POMDP problems. We also compare RLaR against an existing approximate Dec-POMDP solver which, like RLaR, does not assume a priori knowledge of the model. While RLaR's policy representation is not as scalable, we show that RLaR produces higher quality policies for most problems and horizons studied.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shimon Whiteson
- Christian Schröder de Witt
- Tabish Rashid
- Bei Peng
- Chongjie Zhang
- Philip H. S. Torr
- Gregory Farquhar
- Jakob N. Foerster
In Computational Theory and Mathematics
- Shimon Whiteson
- Tabish Rashid
- Christian Schröder de Witt
- Jakob N. Foerster
- Gregory Farquhar
- Mikayel Samvelyan
- Philip H. S. Torr
- Bei Peng
In Reinforcement Learning
- Shimon Whiteson
- Christian Schröder de Witt
- Tabish Rashid
- Chongjie Zhang
- Bei Peng
- Gregory Farquhar
- Jakob N. Foerster
- Philip H. S. Torr
In Game & Decision Theory
- Shimon Whiteson
- Christian Schröder de Witt
- Jakob N. Foerster
- Gregory Farquhar
- Philip H. S. Torr
- Triantafyllos Afouras
- Bei Peng
- Anuj Mahajan
Citations in the corpus
- Counterfactual Multi-Agent Policy Gradients (2017, positive): “…an expectation over R t , V π ( s t ) = E s t +1: ∞ , u t : ∞ [ R t | s t ] , and an action-value function Following previous work (Oliehoek et al., 2008; Kraemer & Banerjee, 2016; Foerster et al., 2016; Jorge et al., 2016), our problem setting allows centralised training but requires…”
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, neutral): “Centralised planning and decentralised execution is also a standard paradigm for multi-agent planning [1, 2].”
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, neutral): “The paradigm of centralised training with decentralised execution (Oliehoek et al., 2008; Kraemer and Banerjee, 2016) has recently attracted attention in the RL community (Jorge et al.”
- Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, neutral): “The framework of centralised training with decentralised execution (CTDE) [17, 14] formalises the use of centralised information to facilitate the training of decentralisable policies.”
- QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, neutral): “IGM (Individual-Global-Max; Son et al., 2019) is a popular principle to realize effective value-based CTDE, which asserts the consistency between joint and local greedy action selections in the joint action-value Qtot(τ ,a) and individual action-values [Qi(τi, ai)] n i=1: ∀τ ∈ T , argmax a∈A Qtot(τ ”
- Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, positive): “We adopt the centralised training and decentralised execution paradigm [20, 14].”
- The StarCraft Multi-Agent Challenge (2019, positive): “This gives rise to the paradigm of centralised training with decentralised execution , which has been well-studied in the planning community (Oliehoek et al., 2008; Kraemer & Banerjee, 2016).”
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, neutral): “The paradigm of centralised training with decentralised execution (Oliehoek et al., 2008; Kraemer & Banerjee, 2016) has recently attracted attention in the RL community (Jorge et al., 2016; Foerster et al., 2018).”
- MAVEN: Multi-Agent Variational Exploration (2019, positive): “As immediate future work, we aim to develop a theoretical analysis similar to QMIX for other CTDE algorithms.”
- Multi-Agent Common Knowledge Reinforcement Learning (2018, neutral): “While significant progress has been made in this direction (Rashid et al., 2018a; Foerster et al., 2016; 2017; 2018; Kraemer & Banerjee, 2016; Jorge et al., 2016), the requirement that policies must be fully decentralised severely limits the agents’ ability to coordinate their behaviour.”
- Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications (2018, unknown): “An alternative and popular approach is the centralized learning and decentralized execution where a group of agents can be trained simultaneously by applying a centralized method via an open communication channel [68].”
- A survey of multi-agent deep reinforcement learning with communication (2022, unknown): “As a balanced solution, Centralized Training and Decentralized Execution (CTDE) [105, 73] has emerged as a popular training schemes in MADRL.”
- FACMAC: Factored Multi-Agent Centralised Policy Gradients (2020, unknown): “They both aim to efficiently learn a centralised but factored action-value function Qπtot, using CTDE.”
- Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning (2020, unknown): “o model these contextual dependencies. To this end, we build our approach on QMIX [27], a state-of-the-art cooperative MARL algorithm in the centralized training with decentralized execution paradigm [26, 18]. QMIX factors the global Qfunction into a non-linear monotonic mixture of individual agent ”
- Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, unknown): “An alternative solution is to employ centralised training with decentralised execution (CTDE Kraemer & Banerjee, 2016).”
- Multi-objective multi-agent decision making: a utility-based analysis and survey (2019, unknown): “The paradigm of centralised training with decentralised execution represents a middle ground between fully centralised and decentralised settings often used in cooperative or mixed settings [53,54,81,86,112,144,156].”