Multi-agent reinforcement learning as a rehearsal for decentralized planning

By Landon Kraemer, Bikramjit Banerjee. 2016

453 citations recorded by the source corpus.

Decentralized partially observable Markov decision processes (Dec-POMDPs) are a powerful tool for modeling multi-agent planning and decision-making under uncertainty. Prevalent Dec-POMDP solution techniques require centralized computation given full knowledge of the underlying model. Multi-agent reinforcement learning (MARL) based approaches have been recently proposed for distributed solution of Dec-POMDPs without full prior knowledge of the model, but these methods assume that conditions during learning and policy execution are identical. In some practical scenarios this may not be the case. We propose a novel MARL approach in which agents are allowed to rehearse with information that will not be available during policy execution. The key is for the agents to learn policies that do not explicitly rely on these rehearsal features. We also establish a weak convergence result for our algorithm, RLaR, demonstrating that RLaR converges in probability when certain conditions are met. We show experimentally that incorporating rehearsal features can enhance the learning rate compared to non-rehearsal-based learners, and demonstrate fast, (near) optimal performance on many existing benchmark Dec-POMDP problems. We also compare RLaR against an existing approximate Dec-POMDP solver which, like RLaR, does not assume a priori knowledge of the model. While RLaR's policy representation is not as scalable, we show that RLaR produces higher quality policies for most problems and horizons studied.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Shimon Whiteson
  2. Christian Schröder de Witt
  3. Tabish Rashid
  4. Bei Peng
  5. Chongjie Zhang
  6. Philip H. S. Torr
  7. Gregory Farquhar
  8. Jakob N. Foerster

In Computational Theory and Mathematics

  1. Shimon Whiteson
  2. Tabish Rashid
  3. Christian Schröder de Witt
  4. Jakob N. Foerster
  5. Gregory Farquhar
  6. Mikayel Samvelyan
  7. Philip H. S. Torr
  8. Bei Peng

In Reinforcement Learning

  1. Shimon Whiteson
  2. Christian Schröder de Witt
  3. Tabish Rashid
  4. Chongjie Zhang
  5. Bei Peng
  6. Gregory Farquhar
  7. Jakob N. Foerster
  8. Philip H. S. Torr

In Game & Decision Theory

  1. Shimon Whiteson
  2. Christian Schröder de Witt
  3. Jakob N. Foerster
  4. Gregory Farquhar
  5. Philip H. S. Torr
  6. Triantafyllos Afouras
  7. Bei Peng
  8. Anuj Mahajan

Citations in the corpus