Leveraging Procedural Generation to Benchmark Reinforcement Learning

By Karl Cobbe, Christopher Hesse, Jacob Hilton, John Schulman. 2019

721 citations recorded by the source corpus.

We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Learning explanations that are hard to vary (2020, background): “The experiments presented in [Cobbe et al., 2019b] showed that state-of-the-art RL algorithms fail to model these invariant mechanisms, performing poorly on new levels unless they are trained on tens of thousands of them.”
Hyperparameter Selection for Imitation Learning (2021, background): “However, RL agents are typically evaluated on their ability to learn on multiple environments (Bellemare et al., 2013; Brockman et al., 2016; Cobbe et al., 2020) with a single set of HPs.”
The MAGICAL Benchmark for Robust Imitation (2020, background): “For instance, there are several challenging benchmarks for generalisation [31, 12, 13] and meta- or multi-task learning [40] in RL.”
Reinforcement Learning with Augmented Data (2020, methodology): “• In the case of Jumper (a navigation task), the gain from data augmentation is not as signiﬁcant because the task involves structural generalization to different map layouts and is likely to require recurrent policies [24].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, neutral): “These include Robot Learning in Homes [21], Meta-World [61], the ProcGen benchmark [10].”
A Generalist Agent (2022, methodology): “We as well include the Procgen Benchmark (Cobbe et al., 2020) and Modular RL (Huang et al.”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “…et al., 2020), due to the difﬁculty in parameterizing different scenes and tasks through a single network (Wang et al., 2018b; Jiang et al., 2019; Zeng et al., 2020) as well as the lack of large off-the-shelf datasets for pre-training (Cobbe et al., 2020; Zhu et al., 2020; Yu et al., 2020).”
Unity: A General Platform for Intelligent Agents (2018, methodology): “Examples of this include the ALE (Bellemare et al., 2013), DMLab-30 (Espeholt et al., 2018), the Hard Eight (Gulcehre et al., 2019), AI2Thor (Kolve et al., 2017), OpenAI Retro (Nichol et al., 2018), DMControl (Tassa et al., 2018b), and ProcGen (Cobbe et al., 2019a).”
Situated Dialogue Learning through Procedural Environment Generation (2021, background): “C L ] 7 O ct 2 02 1 rally generated environments—wherein the agent learns a family of parametrized tasks with a significantly larger state-action spaces than singular environments, thus effectively making the memorization of trajectories impossible (Justesen et al., 2018; Cobbe et al., 2020).”
Mastering Diverse Domains through World Models (2023, methodology): “• ProcGen This benchmark of 16 games features randomized levels and visual distractions to test the robustness and generalization of agents 42 .”
Consistent Dropout for Policy Gradient Reinforcement Learning (2022, background): “Similarly, Cobbe et al. (2018) examined the effects of dropout and `2 regularization on the Procgen benchmark (Cobbe et al., 2019).”
Domain Generalization: A Survey (2021, background): “Reinforcement learning - Coinrun [92] Scenes, difficulty levels - OpenAI Procgen Benchmark [93] States, scenes, rewards, difficulty levels ZHOU ETAL.”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “Procgen (Cobbe et al., 2020) and MiniHack (Küttler et al., 2020; Samvelyan et al., 2021), two of our other benchmarks, are procedurally-generated, like Jelly Bean World (Platanios et al., 2020) which is a procedurally-generated 2D gridworld proposed as a testbed for continual learning agents.”
Prioritized Level Replay (2020, methodology): “…methods for improving generalization in deep RL adapt techniques from supervised learning, including stochastic regularization (Igl et al., 2019; Cobbe et al., 2020a), data augmentation (Kostrikov et al., 2020; Raileanu et al., 2021; Wang et al., 2020a), and feature distillation (Igl et al.,…”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, background): “Indeed, in the past few years a series of challenging PCG environments have been proposed (Juliani et al., 2019; Cobbe et al., 2020; K¨uttler et al., 2020; Samvelyan et al., 2021; Chevalier-Boisvert et al., 2018; Hafner, 2022; Deitke et al., 2022), mostly focusing on testing and improving…”
Increasing generality in machine learning through procedural content generation (2019, result): “In a similar vein to the work by Justesen et al. [4], levels in the CoinRun platform game are procedurally generated to quantify the ability of RL algorithms to generalize to never-before-seen levels [68, 83].”
The NetHack Learning Environment (2020, methodology): “Recent advances in (Deep) Reinforcement Learning (RL) have been driven by the development of novel simulation environments, such as the Arcade Learning Environment (ALE) [9], StarCraft [64, 69], BabyAI [16], Obstacle Tower [38], Minecraft [37, 29, 35], and Procgen Benchmark [18].”