A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

By Amy Zhang, Nicolas Ballas, Joelle Pineau. 2018

200 citations recorded by the source corpus.

The risks and perils of overfitting in machine learning are well known. However most of the treatment of this, including diagnostic tools and remedies, was developed for the supervised learning case. In this work, we aim to offer new perspectives on the characterization and prevention of overfitting in deep Reinforcement Learning (RL) methods, with a particular focus on continuous domains. We examine several aspects, such as how to define and diagnose overfitting in MDPs, and how to reduce risks by injecting sufficient training diversity. This work complements recent findings on the brittleness of deep RL methods and offers practical observations for RL researchers and practitioners.

Subfields: Artificial Intelligence, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Reinforcement Learning

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “Overﬁtting has been observed and studied in RL settings [Zhang et al., 2018a,b, Farebrother et al., 2018, Cobbe et al., 2019].”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, methodology): “We also deﬁne a metric for measuring OOD generalization in our setup called generalization improvement score ( GIS ), inspired by (Zhang et al., 2018).”
Quantifying Generalization in Reinforcement Learning (2018, methodology): “Zhang et al. (2018b) conduct experiments on procedurally generated grid-world mazes, reporting many insightful conclusions on the nature of overﬁtting in RL agents.”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “While previously mentioned studies (Zhang et al., 2018c; Cobbe et al., 2019; Justesen et al., 2018; Juliani et al., 2019) reveal intriguing trends, it is hard to draw general conclusions from so few environments.”
Off-Policy Evaluation via Off-Policy Classification (2019, background): “Several papers [5, 27, 30, 39, 40] have examined overfitting and memorization in deep RL, proposing explicit train-test environment splits as benchmarks for RL generalization.”
Regularization Matters in Policy Optimization (2019, background): “The generalization gap can come from different modes/levels/difficulties of a game (Farebrother et al., 2018), simulation vs. real world (Tobin et al., 2017), parameter variations (Pattanaik et al., 2018), or different random seeds in environment generation (Zhang et al., 2018b).”
Robust Visual Domain Randomization for Reinforcement Learning. (2019, background): “Agents trained on one domain often fail to generalize to other domains that differ only in small ways from the original domain (Sutton, 1996; Cobbe et al., 2019; Zhang et al., 2018b; Packer et al., 2018; Zhang et al., 2018a; Witty et al., 2018; Farebrother et al., 2018).”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, methodology): “Recently, the generalization performance of RL agents has been investigated by splitting training and test environments using random seeds (Zhang et al., 2018a) and distinct sets of levels in video games (Machado et al., 2018; Cobbe et al., 2019).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “This can be done for instance with an encoder-decoder architecture variant (Higgins et al., 2017; Zhang et al., 2018b).”
Replay-Guided Adversarial Environment Design (2021, background): “While deep reinforcement learning (RL) approaches have led to many successful applications in challenging domains like Atari [21], Go [35], Chess [36], Dota [4], and StarCraft [40] in recent years, deep RL agents still prove to be brittle, often failing to transfer to environments only slightly diff”
Towards Better Interpretability in Deep Q-Networks (2018, background): “This is in line with the results of (Zhang et al. 2018) and (Zhang, Ballas, and Pineau 2018).”
An investigation of model-free planning (2019, result): “Generalization in RL is even less well studied, though recent work (Zhang et al., 2018a;b; Cobbe et al., 2018) has begun to explore the effect of training data diversity.”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, background): “Consequently, we argue that this observed degradation of generalisation might contribute to the inferior generalisation properties recently attributed to many RL agents evaluated on held out test environments (Zhang et al., 2018a;b; Zhao et al., 2019).”
Interference and Generalization in Temporal Difference Learning (2020, methodology): “For the generalization experiments of Section 4 we loosely follow the setup of Zhang et al. (2018a): we train RL agents in environments where the initial state is induced by a single random seed, allowing us to have proper training and test sets in the form of mutually exclusive seeds.”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, methodology): “Zhang et al. (2018b) use separate dynamics and reward modules to learn a task embedding.”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, methodology): “…approaches are typically adapted from supervised learning, including regularization (Cobbe et al., 2019; Farebrother et al., 2018), stochasticity (Zhang et al., 2018c), noise injection (Igl et al., 2019; Zhang et al., 2018a), more diverse training conditions (Rajeswaran et al., 2017; Witty et…”
Generalization and Regularization in DQN (2018, background): “Nevertheless, they often investigate generalization in light of diﬀerent aspects of an environment such as noise (e.g., Zhang et al., 2018a) and start state distribution (e.g., Rajeswaran et al., 2017; Zhang et al., 2018a,b).”
Observational Overfitting in Reinforcement Learning (2019, background): “One particular framework, which we denote by zero-shot supervised framework (Zhang et al., 2018a,c; Nichol et al., 2018; Justesen et al., 2018) and is used to study RL generalization, is to treat it analogous to a classical supervised learning (SL) problem – i.e. assume there exists a distribution…”