Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

By Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeff Clune, Kenneth O. Stanley. 2020

155 citations recorded by the source corpus.

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.

Subfields: Artificial Intelligence, Control and Systems Engineering, Mechanical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Mechanical Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2020), and other approaches moving more and more towards open-ended learning (Wang et al., 2020).”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Our procedure for worldagent co-evolution (Section A.1.1) shares similarity with POET (Wang et al., 2019, 2020) and PAIRED (Dennis et al., 2020).”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[13] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth O.”
Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems (2021, background): “Besides, [30] learn an individual policy paired with each task for discovering diverse behaviors while our work learns a single goal-conditioned policy for the entire task space; [32] defines the active set as those parts of the task space in which the value function exhibits the largest gradient w.”
Adversarial Environment Generation for Learning to Navigate the Web (2021, methodology): “To create a curriculum, POET requires generating many new environments, testing all agents within each one, and discarding environments based on a manually chosen a reward threshold, which wastes a signiﬁ-cant amount of computation.”
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (2020, methodology): “While a direct comparison to POET [45] is challenging since many elements of the POET algorithm are speciﬁc to a 2D walker, the main algorithmic distinction between POET and minimax environment design is maintaining a population of environment adversaries and agents that are periodically swapped wit”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “POET and enhanced POET [45, 46] generate the terrain for a 2D walker given access to the environment parameters.”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Prior methods in open-ended agent learning [30, 57, 130, 63, 26] do not make use of external knowledge, but our approach leverages internet-scale database to learn open-vocabulary reward models, thanks to Minecraft’s abundance of gameplay data online.”
Replay-Guided Adversarial Environment Design (2021, methodology): “POET [41, 42], a coevolutionary approach [27], uses a population of minimax (rather than minimax regret) adversaries to construct terrain for a BipedalWalker agent.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Recent works have demonstrated the effectiveness of agent-environment co-adaptation with a distribution of tasks (Wang et al., 2019; Akkaya et al., 2019; Wang et al., 2020a; Parker-Holder et al., 2022).”