Learning with Opponent-Learning Awareness
By Jakob N. Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, Igor Mordatch. 2017
622 citations recorded by the source corpus.
Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical reinforcement learning, generative adversarial networks and decentralised optimisation. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes an additional term that accounts for the impact of one agent's policy on the anticipated parameter update of the other agents. Preliminary results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners' dilemma (IPD), while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to infinitely repeated matching pennies, LOLA agents converge to the Nash equilibrium. In a round robin tournament we show that LOLA agents can successfully shape the learning of a range of multi-agent learning algorithms from literature, resulting in the highest average returns on the IPD. We also show that the LOLA update rule can be efficiently calculated using an extension of the likelihood ratio policy gradient estimator, making the method suitable for model-free reinforcement learning. This method thus scales to large parameter and input spaces and nonlinear function approximators. We also apply LOLA to a grid world task with an embedded social dilemma using deep recurrent policies and opponent modelling. Again, by explicitly considering the learning of the other agent, LOLA agents learn to cooperate out of self-interest.
Subfields: Artificial Intelligence, Management Science and Operations Research, Safety Research, Reinforcement Learning, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
In Management Science and Operations Research
- Sergey Levine
- Shimon Whiteson
- Pieter Abbeel
- David Silver
- Marc G. Bellemare
- R´emi Munos
- Karl Tuyls
- Jakob N. Foerster
In Safety Research
- Adam Lerer
- Alex Peysakhovich
- Pablo Hernandez-Leal
- Bilal Kartal
- Matthew E. Taylor
- Joel Z. Leibo
- Shimon Whiteson
- Jakob N. Foerster
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Shimon Whiteson
- Chelsea Finn
- Yuxi Li
- Marc G. Bellemare
- Doina Precup
- Ofir Nachum
In Game & Decision Theory
- Karl Tuyls
- Julien Pérolat
- Marc Lanctot
- Adam Lerer
- Thore Graepel
- Neil Burch
- Shimon Whiteson
- Jakob N. Foerster
Citations in the corpus
- The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, background): “In decentralized learning [21], each agent optimizes its reward independently; these methods can tackle general-sum games but may suffer from instability even in simple matrix games [12].”
- Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “In recent years, there has been rapid progress on novel MARL algorithms that are based on centralized training with decentralized execution (Self-Play) as a training paradigm (Sunehag et al., 2017; Foerster et al., 2017; Hong et al., 2017; Foerster et al., 2019; Hu & Foerster, 2019).”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…and Faust, 2021b; Kirsch, Flennerhag, van Hasselt, Friesen, Oh, and Chen, 2022; Wan, Peng, and Gangwani, 2022; Melo, 2022; Nam, Sun, Pertsch, Hwang, and Lim, 2022) and multi-agent RL (Foerster et al., 2018a,1; Kim et al., 2021a; Al-Shedivat, Bansal, Burda, Sutskever, Mordatch, and Abbeel, 2017).”
- Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “, 2019) and regularized self-play (Foerster et al., 2018; Perolat et al., 2020; Bai & Jin, 2020).”
- The Hanabi Challenge: A New Frontier for AI Research (2019, background): “ently proposed that learn from models of others whether by using a training regime based on cognitive hierarchies [87], by dening a learning rule that shapes the anticipated learning of other agents [88], or by training architectures that incorporate agent identication as part of the objective and c”
- Mean Field Multi-Agent Reinforcement Learning (2018, background): “Instead, accounting for the extra information from conjec-turing the policies of other agents is beneficial to each single learner (Foerster et al., 2017; Lowe et al., 2017a).”
- Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments (2017, background): “Modeling and exploiting opponents [29, 30] or event their learning processes [31] is advantageous as it improves convergence or helps to discover equilibria of certain properties (e.”
- Emergent Complexity via Multi-Agent Competition (2017, background): “Recently, Foerster et al. [2017b] introduced an algorithm which explicitly accounts for the fact that the opponent is also learning and showed that it can achieve cooperation in iterated prisoner’s dilemma, however the algorithm requires access to the opponent’s parameters.”
- The Mechanics of n-Player Differentiable Games (2018, background): “Learning with opponent-learning awareness (LOLA) infinitesimally modifies the objectives of players to take into account their opponents’ goals (Foerster et al., 2018).”
- Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, background): “…68 ), or non-learning systems ( 50, 69 ), and previously studied multi-agent domains often consist of discrete-state environments ( 18, 38, 64 ), have simplified 2D dynamics ( 23, 41, 53 ) or have fully observable or non-perceptual features ( 18, 23, 41, 48, 53, 64 ) rather than pixel observations.”
- A survey and critique of multiagent deep reinforcement learning (2018, methodology): “Therefore, a LOLA agent directly shapes the policy updates of other agents to maximize its own reward.”
- Relational Forward Models for Multi-Agent Learning (2018, positive): “Despite these recent successes, coordination is still considered a hard problem and several attempts have been made to promote the emergence of coordination by relaxing some assumptions (Foerster et al., 2016; 2017; Sukhbaatar et al., 2016; Raileanu et al., 2018; He et al., 2016a).”
- Emergent Communication through Negotiation (2018, background): “It would be interesting to see whether a learning algorithm, such as Foerster et al. (2017), can discover the same result.”
- Learning to Schedule Communication in Multi-agent Reinforcement Learning (2019, methodology): “To coordinate this exchange of messages, we adopt the centralized training and distributed execution paradigm popularized in recent works, e.g. , Foerster et al. (2017a); Lowe et al. (2017); Sunehag et al. (2018); Rashid et al. (2018); Gupta et al. (2017).”
- Modeling Others using Oneself in Multi-Agent Reinforcement Learning (2018, background): “The closest work to ours is (Foerster et al., 2017) and (He et al., 2016).”
- A Game Theoretic Framework for Model Based Reinforcement Learning (2020, background): “These algorithms are sometimes called independent learners, simultaneous learners, or naive learners [14, 31].”
- Deep Multi Agent Reinforcement Learning for Autonomous Driving (2020, background): “Recent research work [11, 13] has focused on learning decentralized MARL policies by sharing the gradient information between agents during training.”
- Generating Adversarial Driving Scenarios in High-Fidelity Simulators (2019, methodology): “Examples of methods in this category include minimax Qlearning [21], the decentralized Partially Observable Markov Decision Process (POMDP) [22], and more recent multiagent reinforcement learning methods, such as [23], [24].”
- Learning to Teach in Cooperative Multiagent Reinforcement Learning (2018, background): “Similar to recent works that model the multiagent learning process (Hadfield-Menell et al. 2016; Foerster et al. 2018), we focus on the pairwise (two agent) case, targeting the issues of when/what to advise, then detail extensions to n agents.”