Gotta Learn Fast: A New Benchmark for Generalization in RL

By Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, John Schulman. 2018

189 citations recorded by the source corpus.

In this report, we present a new reinforcement learning (RL) benchmark based on the Sonic the Hedgehog (TM) video game franchise. This benchmark is intended to measure the performance of transfer learning and few-shot learning algorithms in the RL domain. We also present and evaluate some baseline algorithms on the new benchmark.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2018) and game levels (Nichol et al., 2018) for training and testing, by exposing internal parameters of various classic environments (Packer et al.”
The MAGICAL Benchmark for Robust Imitation (2020, background): “For instance, there are several challenging benchmarks for generalisation [31, 12, 13] and meta- or multi-task learning [40] in RL.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, methodology): “Meta-reinforcement learning methods have been evaluated on a number of different problems, including maze navigation [18, 19, 20], continuous control domains with parametric variation across tasks [10, 21, 22, 23], bandit problems [19, 18, 20, 24], levels of an arcade game [25], and locomotion tasks”
Quantifying Generalization in Reinforcement Learning (2018, methodology): “Our work is most directly inspired by the Sonic Benchmark (Nichol et al., 2018), which proposes to measure generalization performance by training and testing RL agents on distinct sets of levels in the Sonic the HedgehogTM video game franchise.”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, methodology): “The Sonic benchmark (Nichol et al., 2018) was designed to measure generalization in RL by separating levels of the Sonic the HedgehogTM video game into training and test sets.”
Meta-Learning in Neural Networks: A Survey (2020, background): “Another benchmark [236] is based on splitting Sonic-”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “Several papers [5, 27, 30, 39, 40] have examined overﬁtting and memorization in deep RL, proposing explicit train-test environment splits as benchmarks for RL generalization.”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, background): “There has been some recent attempts in the RL community to study generalization to unseen initial conditions ( e.g. [45, 35]).”
Meta-learning curiosity algorithms (2020, background): “However, as noted by Nichol et al. (2018), Atari games are too different to get big gains with current feature-transfer methods; they instead suggest using different levels of the game Sonic to benchmark generalization.”
Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning (2019, background): “New benchmarks such as OpenAI’s Sonic the Hedgehog emulator and CoinRun inject environmental variation into the training schedule, while introducing train/test splits (Nichol et al., 2018; Cobbe et al., 2018).”
NovelGridworlds: A Benchmark Environment for Detecting and Adapting to Novelties in Open Worlds (2021, background): “The development of environments and benchmarks has helped progress the research in various fields of AI [3, 11, 13, 21, 23].”
Observational Overfitting in Reinforcement Learning (2019, methodology): “One particular framework, which we denote by zero-shot supervised framework (Zhang et al., 2018a,c; Nichol et al., 2018; Justesen et al., 2018) and is used to study RL generalization, is to treat it analogous to a classical supervised learning (SL) problem – i.”
The NetHack Learning Environment (2020, background): “The Gym Retro environment [51] expands the list of classic games, but focuses on evaluating visual generalization and transfer learning on a single game, Sonic The Hedgehog.”
Generalization to New Actions in Reinforcement Learning (2020, background): “Previously, state-space generalization has been used to transfer policies to new environments (Cobbe et al., 2018; Nichol et al., 2018; Packer et al., 2018), agent morphologies (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), and visual inputs for manipulation of unseen…”
Assessing Generalization in Deep Reinforcement Learning (2018, background): “…trained with environmental stochasticity may be more effective for generalization than specialized algorithms; the same conclusion was also suggested by the results of the OpenAI Retro contest (Nichol et al., 2018) and the CoinRun benchmark (Cobbe et al., 2018) in environments with visual input.”
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence (2019, methodology): “Some platforms contain a wide set of diverse games [7, 64, 69, 74]; however, they are designed mostly for single agent scenarios and extremely hard to customize (adding multiple players or creating new games), since the games are provided as compiled binary ROMs.”
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning (2019, methodology): “Other recently introduced game-based AI benchmarks, such as the OpenAI Retro Challenge [Nichol et al., 2018], CoinRun [Cobbe et al.”
Unity: A General Platform for Intelligent Agents (2018, methodology): “Examples of this include the ALE (Bellemare et al., 2013), DMLab-30 (Espeholt et al., 2018), the Hard Eight (Gulcehre et al., 2019), AI2Thor (Kolve et al., 2017), OpenAI Retro (Nichol et al., 2018), DMControl (Tassa et al., 2018b), and ProcGen (Cobbe et al., 2019a).”