Human-level performance in 3D multiplayer games with population-based reinforcement learning

By Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio García Castañeda, Charlie Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman and 8 more. 2018

819 citations recorded by the source corpus.

Artificial teamwork Artificially intelligent agents are getting better and better at two-player games, but most real-world endeavors require teamwork. Jaderberg et al. designed a computer program that excels at playing the video game Quake III Arena in Capture the Flag mode, where two multiplayer teams compete in capturing the flags of the opposing team. The agents were trained by playing thousands of games, gradually learning successful strategies not unlike those favored by their human counterparts. Computer agents competed successfully against humans even when their reaction times were slowed to match those of humans. Science, this issue p. 859 Teams of artificial agents compete successfully against humans in the video game Quake III Arena in Capture the Flag mode. Reinforcement learning (RL) has shown great success in increasingly complex single-agent environments and two-player turn-based games. However, the real world contains multiple agents, each learning and acting independently to cooperate and compete with other agents. We used a tournament-style evaluation to demonstrate that an agent can achieve human-level performance in a three-dimensional multiplayer first-person video game, Quake III Arena in Capture the Flag mode, using only pixels and game points scored as input. We used a two-tier optimization process in which a population of independent RL agents are trained concurrently from thousands of parallel matches on randomly generated environments. Each agent learns its own internal reward signal and rich representation of the world. These results indicate the great potential of multiagent reinforcement learning for artificial intelligence research.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, methodology): “We released the environment to the community as a novel testbed for MARL research.”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Multi-Agent Reinforcement Learning (MARL) Many challenging tasks (Vinyals et al., 2019; Berner et al., 2019; Samvelyan et al., 2019; Jaderberg et al., 2019) can be modelled as multi-agent learning problems.”
Emergent Tool Use From Multi-Agent Autocurricula (2019, methodology): “Our work sees itself in the tradition of previous studies that showcase emergent complexity in simple physically grounded environments [11, 12, 13, 14]; the success in these settings inspires confidence that inducing autocurricula in physically grounded and open-ended environments could eventually e”
Meta-Learning in Neural Networks: A Survey (2020, background): “addressed the single-task setting, where the meta-knowledge such as loss [22], [180], reward [174], [181], hyperparameters [172], [173], or exploration strategy [182] are trained online together with the base policy while learning a single task.”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, background): “…developments and applications to agents that successfully navigate mazes and solve complex tasks from first-person camera observations (Jaderberg et al., 2016; Espeholt et al., 2018; Jaderberg et al., 2019); and robots that successfully grasp objects in the real world (Kalashnikov et al., 2018).”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Like others, we also see multi-agent reinforcement learning as a potential solution to other challenges, such as the design of autocurricula (Leibo et al., 2019) or even end-to-end learning of pixel-perception based agents (Jaderberg et al., 2019).”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, methodology): “Some of the most successful recent applications of deep RL to difﬁcult environments such as Dota 2 (Ope-nAI, 2018a), Capture the Flag (Jaderberg et al., 2019), Starcraft II (Vinyals et al., 2019), and dexterous object manipulation (OpenAI, 2018b) have used policy gradient-based methods such as Proxi”
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance (2019, background): “Recommendations from AI systems—even if imperfect—can result in human-AI teams that perform better than either the human or the AI system alone (Wang et al. 2016; Jaderberg et al. 2019).”
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020, background): “Recent examples using deep learning range from robotic locomotion (Schulman et al., 2015; Gu et al., 2016; Schulman et al., 2017; Levine et al., 2016) to sophisticated video games (Mnih et al., 2013; Schulman et al., 2017; OpenAI, 2018; Jaderberg et al., 2019).”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[14] Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al.”
TarMAC: Targeted Multi-Agent Communication (2018, background): “, 2018), and tasks in novel application domains such as coordination and navigation in 3D simulated environments (Peng et al., 2017; OpenAI, 2018; Jaderberg et al., 2018).”
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning (2020, background): “Population-play: Population-based training is another line of research aiming to improve exploration and coordination in MARL by training a population of diverse sets of agents [35, 17, 16, 19].”