Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
By Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley. 2019
302 citations recorded by the source corpus.
While the history of machine learning so far largely encompasses a series of problems posed by researchers and algorithms that learn their solutions, an important question is whether the problems themselves can be generated by the algorithm at the same time as they are being solved. Such a process would in effect build its own diverse and expanding curricula, and the solutions to problems at various stages would become stepping stones towards solving even more challenging problems later in the process. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper does just that: it pairs the generation of environmental challenges and the optimization of agents to solve those challenges. It simultaneously explores many different paths through the space of possible problems and solutions and, critically, allows these stepping-stone solutions to transfer between problems if better, catalyzing innovation. The term open-ended signifies the intriguing potential for algorithms like POET to continue to create novel and increasingly complex capabilities without bound. Our results show that POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved by direct optimization alone, or even through a direct-path curriculum-building control algorithm introduced to highlight the critical role of open-endedness in solving ambitious challenges. The ability to transfer solutions from one environment to another proves essential to unlocking the full potential of the system as a whole, demonstrating the unpredictable nature of fortuitous stepping stones. We hope that POET will inspire a new push towards open-ended discovery across many domains, where algorithms like POET can blaze a trail through their interesting possible manifestations and solutions.
Subfields: Artificial Intelligence, Reinforcement Learning, Evolutionary Computation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Quoc V. Le
- Chelsea Finn
- Mohit Bansal
- Ruslan Salakhutdinov
- Jianfeng Gao
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Igor Mordatch
- Timothy P. Lillicrap
- Yuxi Li
- Jeff Clune
In Evolutionary Computation
- Jeff Clune
- Sebastian Risi
- Kenneth O. Stanley
- Julian Togelius
- Rui Wang
- Jiale Zhi
- Sergey Levine
- Joel Lehman
Citations in the corpus
- AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “To additionally improve the quality of the search, we allow some workers to search on projected binary MNIST tasks, in addition to projected binary CIFAR-10, to promote diversity (see e.g. (Wang et al., 2019)).”
- Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, methodology): “Unlike methods that adversarially vary the environment [6, 50], our method does not require any additional privileged capability to change the environment parameters.”
- Learning quadrupedal locomotion over challenging terrain (2020, methodology): “We evaluate the traversability of parameterized terrains and use particle filtering to maintain a distribution of terrain parameters of medium difficulty [24, 25] that adapt as the neural network learns.”
- Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[29] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley.”
- Open-ended Learning in Symmetric Zero-sum Games (2019, background): “There is a large literature on novelty search, open-ended evolution, and curiosity, which aim to continually expand the frontiers of game knowledge within an agent (Lehman & Stanley, 2008; Taylor et al., 2016; Banzhaf et al., 2016; Brant & Stanley, 2017; Pathak et al., 2017; Wang et al., 2019).”
- Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning (2020, unknown): “…et al. (2018) apply a curriculum over the control space for robotic manipulation tasks; several works (Wu & Tian, 2016; Pinto et al., 2017; Florensa et al., 2017; Sukhbaatar et al., 2017; Wang et al., 2019) have proposed to first train RL agents on easier goals and switch to harder ones later.”
- MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Natural evolution is able to nurture an ever-expanding tree of diverse life forms thanks to the infinitely varied ecological settings that the Earth supports [102, 114].”
- Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[42] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O.”
- Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning (2021, background): “Previous works have shown the benefits of using an automated curriculum of task difficulty to learn complex locomotion policies [28, 29, 16].”
- Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods (2020, methodology): “Another example is POET (Wang et al. 2019), whose execution could bene-fit from gradually scaling up resources according to the increasing size of active populations in the open-ended search.”
- Replay-Guided Adversarial Environment Design (2021, methodology): “POET [41, 42], a coevolutionary approach [27], uses a population of minimax (rather than minimax regret) adversaries to construct terrain for a BipedalWalker agent.”
- Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions (2020, methodology): “This section first describes the original POET framework (Wang et al., 2019a;b), and then details the two enhancements that help POET reach its potential of producing general open-ended innovation.”
- Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “POET and enhanced POET [45, 46] generate the terrain for a 2D walker given access to the environment parameters.”
- AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, methodology): “That desire motived our recent Paired OpenEnded Trailblazer (POET) algorithm [192].”
- Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, methodology): “While a direct comparison to POET [45] is challenging since many elements of the POET algorithm are specific to a 2D walker, the main algorithmic distinction between POET and minimax environment design is maintaining a population of environment adversaries and agents that are periodically swapped wit”
- Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Recent works have demonstrated the effectiveness of agent-environment co-adaptation with a distribution of tasks (Wang et al., 2019; Akkaya et al., 2019; Wang et al., 2020a; Parker-Holder et al., 2022).”
- Automated curricula through setter-solver interactions (2019, background): “A few studies have considered varying environments; e.g. Wang et al. (2019) considered evolving environments together with paired agents.”
- Increasing generality in machine learning through procedural content generation (2019, methodology): “The POET algorithm ( a ) learns to create increasingly complex environments for a 2D bipedal walker together with their neural network controllers [29].”