Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research

By Joel Z. Leibo, Edward Hughes, Marc Lanctot, Thore Graepel. 2019

125 citations recorded by the source corpus.

Evolution has produced a multi-scale mosaic of interacting adaptive units. Innovations arise when perturbations push parts of the system away from stable equilibria into new regimes where previously well-adapted solutions no longer work. Here we explore the hypothesis that multi-agent systems sometimes display intrinsic dynamics arising from competition and cooperation that provide a naturally emergent curriculum, which we term an autocurriculum. The solution of one social task often begets new social tasks, continually generating novel challenges, and thereby promoting innovation. Under certain conditions these challenges may become increasingly complex over time, demanding that agents accumulate ever more innovations.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[6] Joel Z Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Despite its common usage self-play can be brittle to forgetting past knowledge [180,186,275] (see Sect.”
Inductive biases for deep learning of higher-level cognition (2020, background): “…the non-stationary and competition systems put pressure on the learner to achieve the kind of flexibility, robustness and ability to adapt quickly which humans seem to have when they are faced with new environments (Bansal et al., 2017; Liu et al., 2019; Baker et al., 2019; Leibo et al., 2019).”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “Like others, we also see multi-agent reinforcement learning as a potential solution to other challenges, such as the design of autocurricula (Leibo et al., 2019) or even end-to-end learning of pixel-perception based agents (Jaderberg et al., 2019).”
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, background): “Multi-agent training has been proposed as a way to automatically generate curricula in RL [20, 21, 14].”
Multi-agent deep reinforcement learning: a survey (2021, background): “This process of continual adaption is referred to autocurricula (Leibo et al. 2019), which have been reported in several works (Baker et al. 2020; Sukhbaatar et al. 2017; Svetlik et al. 2017).”
Direct-fit to nature: an evolutionary perspective on biological (and artificial) neural networks (2019, background): “…(e.g., Levine et al., 2018) and adding means to directly interact with other networks (e.g., Goodfellow et al., 2014; Jaderberg et al., 2019) may increase the network’s capacity to learn and reduce the gaps between BNNs and ANNs (Marblestone et al., 2016; Baker et al., 2019; Leibo et al., 2019).”
Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, background): “There are also existing works that study ACL for competitive games [28, 16, 2], which we leave as future work.”
Adversarial Environment Generation for Learning to Navigate the Web (2021, methodology): “Multi-agent training can be an effective method for automatically generating a curriculum of RL tasks (e.g. Leibo et al. (2019 For example, Asymmetric Self Play (ASP) (Sukhbaatar et al., 2017) trains two agents, in which the second agent must learn to repeat the actions taken by the ﬁrst,…”
From Motor Control to Team Play in Simulated Humanoid Football (2021, background): “Competitive play within a population, combined with a mechanism to propagate high-performing policies through the population, induces an autocurriculum in which environment difficulty (determined by the strength of opponents in the population) is effectively calibrated to a practical but challenging”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, background): “Modern learning algorithms are more of outstanding test-takers, but less of innovators [4], i.e., the ceiling of an agent’s intelligence may be limited by the complexity of its environment [52].”