Emergent Complexity via Multi-Agent Competition
By Trapit Bansal, Jakub W. Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch. 2017
435 citations recorded by the source corpus.
Reinforcement learning algorithms can train agents that solve problems in complex, interesting environments. Normally, the complexity of the trained agent is closely related to the complexity of the environment. This suggests that a highly capable agent requires a complex environment for training. In this paper, we point out that a competitive multi-agent environment trained with self-play can produce behaviors that are far more complex than the environment itself. We also point out that such environments come with a natural curriculum, because for any skill level, an environment full of agents of this level will have the right level of difficulty. This work introduces several competitive multi-agent environments where agents compete in a 3D world with simulated physics. The trained agents learn a wide variety of complex and interesting skills, even though the environment themselves are relatively simple. The skills include behaviors such as running, blocking, ducking, tackling, fooling opponents, kicking, and defending using both arms and legs. A highlight of the learned behaviors can be found here: this https URL
Subfields: Artificial Intelligence, Reinforcement Learning, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Chelsea Finn
- Jianfeng Gao
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Nicolas Manfred Otto Heess
- Yuxi Li
- Ofir Nachum
- Shixiang Shane Gu
In Game & Decision Theory
- Karl Tuyls
- Julien Pérolat
- Marc Lanctot
- Zhuoran Yang
- Thore Graepel
- Pablo Hernandez-Leal
- Matthew E. Taylor
- David Silver
Citations in the corpus
- Wireless Network Intelligence at the Edge (2018, methodology): “where each agent’s policy is trained using SGD with an NN [188].”
- Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “Self-play was shown to be a powerful algorithm for learning skills within high-dimensional continuous environments [33] and a method for automatically generating curricula [34].”
- Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “To encourage faster learning, as in (Al-Shedivat et al., 2018) and (Bansal et al., 2018), we also trained all agents using a dense reward curriculum in the initial stages of training.”
- Maximum a Posteriori Policy Optimisation (2018, background): “, 2016), multi-agent behaviour (Bansal et al., 2017) and classical control (Duan et al.”
- Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments [11, 12, 13, 14]; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually e”
- Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments (2017, methodology): “nsure consistency of the training curriculum for all agents, we first pre-train a number of policies of each type for every agent type via pure self-play with the PPO algorithm (Schulman et al., 2017; Bansal et al., 2018). We snapshot and save versions of the pretrained policies at each iteration. Th”
- A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning (2017, background): “The generality offered by PSRO/DCH can be seen as a form of “opponent/teammate regularization”, and has also been observed recently in practice [66, 5].”
- Human-level performance in 3D multiplayer games with population-based reinforcement learning (2018, neutral): “While self-play variants can prove effective in some multi-agent games ( 4, 9, 24, 37, 47, 57, 58 ), these methods can be unstable and in their basic form do not support concurrent training which is crucial for scalability.”
- A survey and critique of multiagent deep reinforcement learning (2018, methodology): “, evolutionary methods [20,85,185,271] or sampling-based methods, have shown good results [24,156,187].”
- Relational Forward Models for Multi-Agent Learning (2018, positive): “…where multiple learning agents perceive and act in a shared environment, has produced impressive results (Jaderberg et al., 2018; Pachocki et al., 2018; Leibo et al., 2017; Hughes et al., 2018; Peysakhovich & Lerer, 2017a; Lerer & Peysakhovich, 2017; Bansal et al., 2017; Lanctot et al., 2017).”
- Inductive biases for deep learning of higher-level cognition (2020, background): “At the same time, we make the case that evolutionary forces, the interactions between multiple agents, the non-stationary and competition systems put pressure on the learner to achieve the kind of flexibility, robustness and ability to adapt quickly which humans seem to have when they are faced with”
- Emergent Coordination Through Competition (2019, background): “Bansal et al. (2017); Al-Shedivat et al. (2017) introduce a MuJoCo Sumo domain with similar motivation to ours, and observe emergent complexity from competition, in a 1v1 domain.”
- Open-ended Learning in Symmetric Zero-sum Games (2019, background): “There are very successful examples of algorithms that pose and solve a series of increasingly difficult problems for themselves through forms of self-play (Silver et al., 2018; Jaderberg et al., 2018; Bansal et al., 2018; Tesauro, 1995).”
- CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning (2018, positive): “While this decomposition is implicitly available in a wide range of complex multi-agent environments (Bansal et al., 2018; Foerster et al., 2018; Lowe et al., 2017; Rashid et al., 2018; Liu et al., 2019; Jaderberg et al., 2019), we explicitly use it to implement our curriculum.”