Emergent Coordination Through Competition

By Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Manfred Otto Heess, Thore Graepel. 2019

161 citations recorded by the source corpus.

We study the emergence of cooperative behaviors in reinforcement learning agents by introducing a challenging competitive multi-agent soccer environment with continuous simulated physics. We demonstrate that decentralized, population-based training with co-play can lead to a progression in agents' behaviors: from random, to simple ball chasing, and finally showing evidence of cooperation. Our study highlights several of the challenges encountered in large scale multi-agent training in continuous control. In particular, we demonstrate that the automatic optimization of simple shaping rewards, not themselves conducive to co-operative behavior, can lead to long-horizon team behavior. We further apply an evaluation scheme, grounded by game theoretic principals, that can assess agent performance in the absence of pre-defined evaluation tasks or human baselines.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, background): “Many researchers have developed agents for other football games (e.g., RoboCup Soccer Simulator (Kitano et al., 1997) and the DeepMind MuJoCo Multi-Agent Soccer Environment (Liu et al., 2019, 2021a)).”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments [11, 12, 13, 14]; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually e”
Google Research Football: A Novel Reinforcement Learning Environment (2019, methodology): “There are other available football simulators, such as the RoboCup Soccer Simulator (Kitano et al. 1995; 1997), and the DeepMind MuJoCo Multi-Agent Soccer Environment (Liu et al. 2019).”
Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery (2019, background): “While cooperative behavior may emerge from flat policies trained by a team reward [21], explicitly training high-level skill-selection policies allows external control over the choice of skills performed (by fixing a latent variable), and subsequent analysis of the behavior for each skill.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “• MuJoCo Multiagent Soccer [313] uses the MuJoCo physics engine [202].”
Inductive biases for deep learning of higher-level cognition (2020, background): “At the same time, we make the case that evolutionary forces, the interactions between multiple agents, the non-stationary and competition systems put pressure on the learner to achieve the kind of flexibility, robustness and ability to adapt quickly which humans seem to have when they are faced with”
Multi-agent deep reinforcement learning: a survey (2021, background): “A different research direction can be seen in distributed population-based training schemes where agents are optimized through an online evolutionary process such that under-performing agents are substituted by mutated versions of better agents (Jaderberg et al. 2019; Liu et al. 2019).”
Deep Multi-Agent Reinforcement Learning for Decentralized Continuous Cooperative Control (2020, background): “Although some multi-agent benchmark environments for continuous control exist (Leibo et al., 2017; Liu et al., 2019), few environments specialise in cooperative control and even fewer model partial observability.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “, 2019) or uniformly sampled tasks (Baker et al., 2020; Cultural General Intelligence Team et al., 2022; Jaderberg et al., 2019; Liu et al., 2019).”
Intrinsic Motivation for Encouraging Synergistic Behavior (2020, background): “The idea of motivating an agent to reach areas of the state space which yield high model prediction error was first proposed by Schmidhuber (1991). Generally, this reward obeys the form ‖f(x)− f̂(x)‖, i.”
Multi-objective multi-agent decision making: a utility-based analysis and survey (2019, background): “[85] study how coordination can emerge in a competitive setting, through reward shaping.”
Effective Diversity in Population-Based Reinforcement Learning (2020, methodology): “Population-based methods are particularly prominent in the Neuroevolution community (Stanley et al., 2019), but have recently been of increased interest in RL (Jung et al., 2020; Liu et al., 2019; Khadka and Tumer, 2018; Jung et al., 2020; Khadka et al., 2019; Pourchot and Sigaud, 2019).”
Emergence of Scenario-Appropriate Collaborative Behaviors for Teams of Robotic Bodyguards (2018, background): “Experimental data (see Section 4.2) conﬁrm that na¨ıve MARL scenarios trained on speciﬁc environments perform poorly in different environments.”
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition (2021, background): “Liu et al. (2019) and Jaderberg et al. (2019) combined population-based training with self-play to solve complex team competition tasks, a popular 3 D multiplayer ﬁrst-person video game, Quake III Arena Capture the Flag , and MuJoCo Soccer , respectively.”
PettingZoo: Gym for Multi-Agent Reinforcement Learning (2020, unknown): “This design choice has become the standard for MARL outside of strictly turn-based games like poker, where simultaneous stepping would be a poor conceptual ﬁt [Lowe et al., 2017, Zheng et al., 2017, Gupta et al., 2017, Liu et al., 2019, Liang et al., 2018, Weng et al., 2020].”
α-Rank: Multi-Agent Evaluation by Evolution (2019, methodology): “Finally, we evaluated the approach on an extensive range of meta-game domains including AlphaGo [1], AlphaZero [3], MuJoCo Soccer [5], and Poker [29], which exhibit a range of complexities in terms of payoff asymmetries, number of players, and number of agents involved.”
Meta reinforcement learning as task inference (2019, methodology): “We use the same distributed setup and strategy for initializing recurrent networks during off-policy learning as in Liu et al. (2019).”