Quantifying Generalization in Reinforcement Learning

By Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, John Schulman. 2018

782 citations recorded by the source corpus.

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “Overﬁtting has been observed and studied in RL settings [Zhang et al., 2018a,b, Farebrother et al., 2018, Cobbe et al., 2019].”
Unsolved Problems in ML Safety (2021, background): “Interactive environments [37] could simulate qualitatively distinct random shocks that irreversibly shape the environment’s future evolution.”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “…frederik.traeuble@tuebingen.mpg.de 4 https://sites.google.com/view/causal-world/home In existing benchmarks (Yu et al., 2019; Goyal et al., 2019a; Cobbe et al., 2018; Bellemare et al., 2013; James et al., 2020) the amount of shared causal structure between the different environments is…”
Learning explanations that are hard to vary (2020, methodology): “Lastly, we set up a behavioural cloning task based on the game CoinRun [Cobbe et al., 2018], and observe better generalization on new unseen levels.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Recently, Coinrun (Cobbe et al., 2019) was proposed that is a procedurally generated environment having different training and testing sets to measure generalization in RL.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2018), and by procedurally generating environments (Justesen et al., 2018; Zhang et al., 2018a; Cobbe et al., 2018).”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “CoinRun [10] 64×64×3 no Randomly generated maps with 3 levels of difficulty.”
MOPO: Model-based Offline Policy Optimization (2020, background): “However, such online data collection can incur high sample complexity [45, 55, 56], limit the power of generalization to unseen random initialization [8, 74, 4], and pose risks in safety-critical settings [67].”
The MAGICAL Benchmark for Robust Imitation (2020, background): “For instance, there are several challenging benchmarks for generalisation [31, 12, 13] and meta- or multi-task learning [40] in RL.”
Reinforcement Learning with Augmented Data (2020, methodology): “Cobbe et al. [36] and Lee et al. [37] showed that simple data augmentation techniques such as cutout [36] and random convolution [37] can be useful to improve generalization of agents on the OpenAI CoinRun and ProcGen benchmarks.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, neutral): “Prior work has explored regularization of the neural network approximator in deep RL, e.g. using dropout [17] and cutout [11] techniques.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, background): “Previous works that have proposed benchmarks for reinforcement learning have largely focused on single task learning settings [11, 12, 13].”
Lifelong Generative Modeling (2017, background): “This contrasts the state of the art machine learning models described above which use hundreds of thousands of samples and fail to generalize to slight variations of the original task [22].”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, methodology): “While previously mentioned studies (Zhang et al., 2018c; Cobbe et al., 2019; Justesen et al., 2018; Juliani et al., 2019) reveal intriguing trends, it is hard to draw general conclusions from so few environments.”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “[226] proposed two purpose designed video games for benchmarking Meta-RL.”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “Several papers [5, 27, 30, 39, 40] have examined overﬁtting and memorization in deep RL, proposing explicit train-test environment splits as benchmarks for RL generalization.”
Generalization to New Actions in Reinforcement Learning (2020, methodology): “Previously, state-space generalization has been used to transfer policies to new environments (Cobbe et al., 2018; Nichol et al., 2018; Packer et al., 2018), agent morphologies (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), and visual inputs for manipulation of unseen…”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “However, recent studies have pointed out that RL agents trained and tested on the same environment tend to overfit to that environment’s idiosyncracies and are unable to generalize to even small perturbations (Whiteson et al., 2011; Rajeswaran et al., 2017; Zhang et al., 2018c;a; Henderson et al., 2”
Options as responses: Grounding behavioural hierarchies in multi-agent RL (2019, background): “Consequently, researchers have begun advocating borrowing experimental design from supervised learning with explicitly separated testing and training version of the environment (Machado et al., 2018; Cobbe et al., 2019; Racanière et al., 2017).”