Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

By Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M. Bayen, Stuart J. Russell, Andrew Critch, Sergey Levine. Neural Information Processing Systems, 2020

327 citations recorded by the source corpus.

A wide range of reinforcement learning (RL) problems - including robustness, transfer learning, unsupervised RL, and emergent complexity - require specifying a distribution of tasks or environments in which a policy will be trained. However, creating a useful distribution of environments is error prone, and takes a significant amount of developer time and effort. We propose Unsupervised Environment Design (UED) as an alternative paradigm, where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. Existing approaches to automatically generating environments suffer from common failure modes: domain randomization cannot generate structure or adapt the difficulty of the environment to the agent's learning progress, and minimax adversarial training leads to worst-case environments that are often unsolvable. To generate structured, solvable environments for our protagonist agent, we introduce a second, antagonist agent that is allied with the environment-generating adversary. The adversary is motivated to generate environments which maximize regret, defined as the difference between the protagonist and antagonist agent's return. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/93b2788fb1f2aed0e545d9f9d7dca1c05a63208a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Adversarial Environment Generation for Learning to Navigate the Web (2021, methodology): “The regret is then estimated as: As Dennis et al. (2020) show, if the adversary and antagonist coordinate and reach a Nash equilibrium with the protagonist, then the protagonist will have learned to minimize the regret.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “A pi-oneering method in this space was PAIRED (Dennis et al., 2020), which generates tasks using an RL-trained adversary.”
Prioritized Level Replay (2020, background): “Prior works have shown that directly altering levels to match their difﬁculty to the agent’s abilities can improve generalization (Justesen et al., 2018; Dennis et al., 2020; Chevalier-Boisvert et al., 2018; Zhong et al., 2020).”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, methodology): “3 [29] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine.”
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents (2023, background): “To fully see the difference between humans and models, we select the most challenging scenarios following Dennis et al. (2020); Swayamdipta et al. (2020).”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “[15] Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre M.”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “, 2020)—methods either choose to adversarially set goals of steadily increasing difficulty (Sukhbaatar et al., 2018; Racaniere et al., 2019; Dennis et al., 2020; Campero et al., 2021) or to maximize learning performance based on environment instances an agent finds difficult historically (Graves et ”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Our procedure for worldagent co-evolution (Section A.1.1) shares similarity with POET (Wang et al., 2019, 2020) and PAIRED (Dennis et al., 2020).”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, methodology): “Prior methods in open-ended agent learning [30, 57, 130, 63, 26] do not make use of external knowledge, but our approach leverages internet-scale database to learn open-vocabulary reward models, thanks to Minecraft’s abundance of gameplay data online.”
Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks (2023, background): “For example, Dennis et al. [5] generated a natural curriculum of increasingly complex environments and Parker-Holder et al. [16] harnessed the power of evolution in a principled, regret-based curriculum.”