POET: open-ended coevolution of environments and their optimized solutions

By Rui Wang, Joel Lehman, Jeff Clune, Kenneth O. Stanley. 2019

103 citations recorded by the source corpus.

How can progress in machine learning and reinforcement learning be automated to generate its own never-ending curriculum of challenges without human intervention? The recent emergence of quality diversity (QD) algorithms offers a glimpse of the potential for such continual open-ended invention. For example, novelty search showcases the benefits of explicit novelty pressure, MAP-Elites and Innovation Engines highlight the advantage of explicit elitism within niches in an otherwise divergent process, and minimal criterion coevolution (MCC) reveals that problems and solutions can coevolve divergently. The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper combines these principles to produce a practical approach to generating an endless progression of diverse and increasingly challenging environments while at the same time explicitly optimizing their solutions. An intriguing implication is the opportunity to transfer solutions among environments, reflecting the view that innovation is a circuitous and unpredictable process. POET is tested in a 2-D obstacles course domain, where it generates diverse and sophisticated behaviors that create and solve a wide range of environmental challenges, many of which cannot be solved by direct optimization, or by a direct-path curriculum-building control algorithm. We hope that POET will inspire a new push towards open-ended discovery across many domains.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, methodology): “In similar spirit to our ADR, related works include PowerPlay [97, 103], POET [117] and different varieties of self-play [101, 102, 76].”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Our procedure for worldagent co-evolution (Section A.1.1) shares similarity with POET (Wang et al., 2019, 2020) and PAIRED (Dennis et al., 2020).”
Text Data Augmentation for Deep Learning (2021, methodology): “This is a very interesting idea encompassing papers such as POET [11], Generative Teaching Networks [12], and the Synthetic Petri Dish [13] which describe algorithms that learn the environment to learn from.”
Prioritized Level Replay (2020, methodology): “A recent theme in the PCG setting explores adaptively generating levels to facilitate learning (Sukhbaatar et al., 2017; Wang et al., 2019; 2020b; Khalifa et al., 2020; Akkaya et al., 2019; Campero et al., 2020; Dennis et al., 2020).”
Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions (2020, methodology): “This section ﬁrst describes the original POET framework (Wang et al., 2019a;b), and then details the two enhancements that help POET reach its potential of producing general open-ended innovation.”