Multiagent cooperation and competition with deep reinforcement learning
By Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, Raul Vicente. 2015
1,014 citations recorded by the source corpus.
Evolution of cooperation and competition can appear when multiple adaptive agents share a biological, social, or technological niche. In the present work we study how cooperation and competition emerge between autonomous agents that learn by reinforcement while using only their raw visual input as the state representation. In particular, we extend the Deep Q-Learning framework to multiagent environments to investigate the interaction between two learning agents in the well-known video game Pong. By manipulating the classical rewarding scheme of Pong we show how competitive and collaborative behaviors emerge. We also describe the progression from competitive to collaborative behavior when the incentive to cooperate is increased. Finally we show how learning by playing against another adaptive agent, instead of against a hard-wired algorithm, results in more robust strategies. The present work shows that Deep Q-Networks can become a useful tool for studying decentralized learning of multiagent systems coping with high-dimensional environments.
Subfields: Artificial Intelligence, Sociology and Political Science, Statistical and Nonlinear Physics, Reinforcement Learning, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Shimon Whiteson
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Pieter Abbeel
- Percy Liang
- Jianfeng Gao
- Ruslan Salakhutdinov
In Sociology and Political Science
- Shimon Whiteson
- Alex Peysakhovich
- Adam Lerer
- Thore Graepel
- Karl Tuyls
- Christian Schröder de Witt
- Chongjie Zhang
- Tonghan Wang
In Statistical and Nonlinear Physics
- Ying Nian Wu
- Song-Chun Zhu
- Josh Tenenbaum
- Jianyi Wang
- Pieter Abbeel
- Rob Fergus
- Igor Mordatch
- Pushmeet Kohli
In Reinforcement Learning
- Shimon Whiteson
- Chongjie Zhang
- Sergey Levine
- Jakob N. Foerster
- Tonghan Wang
- Tabish Rashid
- Christian Schröder de Witt
- Pieter Abbeel
In Game & Decision Theory
- Shimon Whiteson
- Karl Tuyls
- Julien Pérolat
- Adam Lerer
- Alex Peysakhovich
- Thore Graepel
- Marc Lanctot
- Pablo Hernandez-Leal
Citations in the corpus
- Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, methodology): “[14] address this setting with a framework that combines DQN with independent Q-learning, in which each agent a independently and simultaneously learns its own Q-function Q(s, u; θ i ).”
- Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, methodology): “While some multi-agent testbeds have emerged, such as Poker (Heinrich and Silver, 2016), Pong (Tampuu et al., 2017), Keepaway Soccer (Stone et al., 2005), or simple gridworld-like environments (Lowe et al., 2017; Leibo et al., 2017; Yang et al., 2018; Zheng et al., 2017), there are currently no…”
- Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (2017, background): “Deep Q-learning approaches have previously been investigated in [33] to train competing Pong agents.”
- Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, methodology): “Tampuu et al. (2015) apply a framework that combines DQN with independent Q-learning to two-player pong.”
- Learning Multiagent Communication with Backpropagation (2016, background): “Sev ral recent works have applied th se methods t multi-agent domains, such s Go [16, 24] and Atari games [29], but t ey assume fu l visibility of the e vir nm t and lack comm ni ati n.”
- Weighted QMIX: Expanding Monotonic Value Function Factorisation (2020, methodology): “Tampuu et al. [30] train each agent’s DQN [19] using Independent Q -learning [31], which treats the other agents as part of the environment, which can lead to many pathologies and instability during training [4, 8].”
- QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning (2019, methodology): “Tampuu et al. (2017) combines this algorithm with deep learning techniques presented in DQN (Mnih et al., 2015).”
- The StarCraft Multi-Agent Challenge (2019, neutral): “Some testbeds have emerged for other multi-agent regimes, such as Poker (Heinrich & Silver, 2016), Pong (Tampuu et al., 2015), Keepaway Soccer(Stone et al.”
- QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2018, methodology): “(Tampuu et al., 2017) extend this approach to deep neural networks using DQN (Mnih et al., 2015).”
- MAVEN: Multi-Agent Variational Exploration (2019, methodology): “In CTDE [26], [47] extend Independent Q-Learning [48] to use DQN to learn Q-values for each agent independently.”
- Learning Nearly Decomposable Value Functions Via Communication Minimization (2019, background): “On the other hand, directly learning decentralized action-value function Q i for each agent alleviates the scalability problem (Tan, 1993; Tampuu et al., 2017).”
- From Few to More: Large-scale Dynamic Multiagent Curriculum Learning (2019, methodology): “Figure5(a-c) show the average win rate of IQL with and without our DyMA-CL under different network structures (i.e., Vanilla network does not contain the GNN part, and MAX means the GNN uses MAX as the aggregation operator).”
- Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control (2019, methodology): “Although IQL is successful in solving simple tasks such as Pong [19], it ignores the environment dynamics arose from the interactions among the agents.”
- The Representational Capacity of Action-Value Networks for Multi-Agent Reinforcement Learning (2019, background): “This suggests that such approaches are a promising direction forward for scalable deep MARL in many problem settings.”
- Actor-Attention-Critic for Multi-Agent Reinforcement Learning (2018, methodology): “As deep learning based approaches to reinforcement learning have grown more popular, they have, naturally, been applied to the MARL setting (Tampuu et al., 2017; Gupta et al., 2017), allowing multi-agent learning in highdimensional/continuous state spaces; however, naive applications of Deep RL meth”
- Lenient Multi-Agent Deep Reinforcement Learning (2017, background): “Past research has shown that deep reinforcement learning agents can converge towards cooperative policies in domains where the agents receive feedback for their individual actions, such as when learning to play pong with the goal of keeping the ball in play for as long as possible [28].”