A Policy Gradient Algorithm for Learning to Learn in Multiagent Reinforcement Learning

By Dong-Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sebastian Lopez-Cot, G. Tesauro, Jonathan P. How. 2020

69 citations recorded by the source corpus.

A fundamental challenge in multiagent reinforcement learning is to learn beneficial behaviors in a shared environment with other agents that are also simultaneously learning. In particular, each agent perceives the environment as effectively non-stationary due to the changing policies of other agents. Moreover, each agent is itself constantly learning, leading to natural nonstationarity in the distribution of experiences encountered. In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accommodates for the non-stationary policy dynamics inherent to these multiagent settings. This is achieved by modeling our gradient updates to directly consider both an agent's own non-stationary policy dynamics and the non-stationary policy dynamics of other agents interacting with it in the environment. We find that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently combines key aspects of previous state of the art approaches on this topic. We test our method on several multiagent benchmarks and demonstrate a more efficient ability to adapt to new agents as they learn than previous related approaches across the spectrum of mixed incentive, competitive, and cooperative environments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, in multi-agent RL an agent may be able to influence the learned behavior of another agent by changing its own behavior and play an active role in shaping the non-stationarity of the RL problem from its perspective (Foerster et al., 2018a,b; Kim et al., 2020).”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “While the objective in Equation (3) ideally maximizes over the space of update functions and learns a non-stationary policy, addressing the computational difficulty of long horizon meta-learning still remains an active area of research [9, 29, 30].”