Emergent Tool Use From Multi-Agent Autocurricula

By Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch. 2019

768 citations recorded by the source corpus.

Through multi-agent competition, the simple objective of hide-and-seek, and standard reinforcement learning algorithms at scale, we find that agents create a self-supervised autocurriculum inducing multiple distinct rounds of emergent strategy, many of which require sophisticated tool use and coordination. We find clear evidence of six emergent phases in agent strategy in our environment, each of which creates a new pressure for the opposing team to adapt; for instance, agents learn to build multi-object shelters using moveable boxes which in turn leads to agents discovering that they can overcome obstacles using ramps. We further provide evidence that multi-agent competition may scale better with increasing environment complexity and leads to behavior that centers around far more human-relevant skills than other self-supervised reinforcement learning methods such as intrinsic motivation. Finally, we propose transfer and fine-tuning as a way to quantitatively evaluate targeted capabilities, and we compare hide-and-seek agents to both intrinsic motivation and random initialization baselines in a suite of domain-specific intelligence tests.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Baker et al. (2020) exhibit reward hacking in a hide-and-seek environment comprising 3-6 agents, 3-9 movable boxes and a few ramps: without a penalty for leaving the play area, the hiding agents learn to endlessly run from the seeking agents.”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, background): “Recent advances in reinforcement learning (RL) and multi-agent reinforcement learning (MARL) have led to a great deal of progress in creating artificial agents which can cooperate to solve tasks: DeepMind’s AlphaStar surpassed professional-level performance in the StarCraft II [34], OpenAI Five defe”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “…in domains such as game playing (Mnih et al., 2013; Silver et al., 2017), learning motor control for high-dimensional simulated robots (Akkaya et al., 2019), multi-agent settings (Baker et al., 2019; Berner et al., 2019) and for studying transfer in the context of meta-learning (Yu et al., 2019).”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “[43, 4] advocate the need to develop CL methods suitable for reinforcement learning training as a necessary step towards learning artiﬁcial intelligent agents to operate in open-ended and changing environments.”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Adversarial games have been proposed in RL with the goal of improving exploration [42, 27], learning goal conditioned policies [36] and generating an automated curriculum [45, 41, 2].”
RODE: Learning Roles to Decompose Multi-Agent Tasks (2020, background): “Many interesting real-world phenomena, including the emergence of tool usage (Baker et al., 2020), communication (Foerster et al., 2016; Sukhbaatar et al., 2016; Lazaridou et al., 2017; Das et al., 2019), social inﬂuence (Jaques et al., 2019), and inequity aversion (Hughes et al., 2018) can also be…”
ROMA: Multi-Agent Reinforcement Learning with Emergent Roles (2020, background): “, 2020), the emergence of fairness (Jiang & Lu, 2019), and the emergence of tool usage (Baker et al., 2020) provide a deep learning perspective in understanding both natural and artificial multi-agent systems.”
Inductive biases for deep learning of higher-level cognition (2020, background): “…the non-stationary and competition systems put pressure on the learner to achieve the kind of flexibility, robustness and ability to adapt quickly which humans seem to have when they are faced with new environments (Bansal et al., 2017; Liu et al., 2019; Baker et al., 2019; Leibo et al., 2019).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “In addition, we see the agent being capable in tasks that not only are explicitly held-out from training, but lie far outside of its training distributions, including versions of hide and seek (Baker et al., 2020) and capture the flag (Jaderberg et al., 2019).”
Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, positive): “Baker et al. (2019) show that multi-agent self-play can also lead to autocurricula in open-ended environments.”
Unity: A General Platform for Intelligent Agents (2018, background): “The features of the toolkit include a set of example environments, state of the art RL algorithms Soft Actor-Critic (SAC) (Haarnoja et al., 2018) and Proximal Policy Optimization (PPO) (Schulman et al., 2017), the IL algorithms Generative Adversarial Imitation Learning (GAIL) (Ho and Ermon, 2016) an”
Near-Optimal Reinforcement Learning with Self-Play (2020, background): “Multi-agent RL has achieved significant recent success on traditionally challenging tasks, for example in the game of GO [30, 31], Poker [6], real-time strategy games [33, 22], decentralized controls or multiagent robotics systems [5], autonomous driving [27], as well as complex social scenarios suc”
A Sharp Analysis of Model-based Reinforcement Learning with Self-Play (2020, background): “, 2019), as well as behavior learning in complex social scenarios (Baker et al., 2020).”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, result): “Similar results have been shown in the context of RL agents by Baker et al. (2020). They demonstrated that increasingly complex behaviors can emerge in a physically grounded task.”
Joint Policy Search for Multi-agent Collaboration with Imperfect Information (2020, methodology): “Self-play was used in large-scale IG that requires collaboration like Dota 2 [30] and Find and Seek [1].”
SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments (2021, background): “Existing work has also studied how competitive self-play and competitive, multi-agent environments can lead to complex behaviors with minimal reward information (Silver et al., 2017; Bansal et al., 2017; Sukhbaatar et al., 2017; Baker et al., 2019; Weihs et al., 2019; Chen et al., 2020).”
Tool Learning with Foundation Models (2023, neutral): “, 2018) and multi-agent autocurricula (Baker et al., 2020).”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, methodology): “Although simulation-based multi-agent environments [3,59,6] can partially address such an issue, learning from noisy and real visual input in physical work is still essential for understanding collaborative planning behaviors of agents in the context of complex daily tasks.”