Learning with Opponent-Learning Awareness

By Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, Igor Mordatch. 2017

622 citations recorded by the source corpus.

Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical reinforcement learning, generative adversarial networks and decentralised optimisation. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes an additional term that accounts for the impact of one agent's policy on the anticipated parameter update of the other agents. Preliminary results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners' dilemma (IPD), while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to infinitely repeated matching pennies, LOLA agents converge to the Nash equilibrium. In a round robin tournament we show that LOLA agents can successfully shape the learning of a range of multi-agent learning algorithms from literature, resulting in the highest average returns on the IPD. We also show that the LOLA update rule can be efficiently calculated using an extension of the likelihood ratio policy gradient estimator, making the method suitable for model-free reinforcement learning. This method thus scales to large parameter and input spaces and nonlinear function approximators. We also apply LOLA to a grid world task with an embedded social dilemma using deep recurrent policies and opponent modelling. Again, by explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest.

Subfields: Artificial Intelligence, Management Science and Operations Research, Safety Research, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Pieter Abbeel
  4. Mohit Bansal
  5. Ruslan Salakhutdinov
  6. Quoc V. Le
  7. Jianfeng Gao
  8. Percy Liang

In Management Science and Operations Research

  1. Sergey Levine
  2. Shimon Whiteson
  3. Pieter Abbeel
  4. David Silver
  5. Marc G. Bellemare
  6. R´emi Munos
  7. Karl Tuyls
  8. Jakob N. Foerster

In Safety Research

  1. Adam Lerer
  2. Alex Peysakhovich
  3. Pablo Hernandez-Leal
  4. Bilal Kartal
  5. Matthew E. Taylor
  6. Joel Z. Leibo
  7. Shimon Whiteson
  8. Jakob N. Foerster

In Reinforcement Learning

  1. Sergey Levine
  2. Pieter Abbeel
  3. Shimon Whiteson
  4. Chelsea Finn
  5. Yuxi Li
  6. Marc G. Bellemare
  7. Doina Precup
  8. Ofir Nachum

In Game & Decision Theory

  1. Karl Tuyls
  2. Julien Pérolat
  3. Marc Lanctot
  4. Adam Lerer
  5. Thore Graepel
  6. Neil Burch
  7. Shimon Whiteson
  8. Jakob N. Foerster

Citations in the corpus