A Study on Overfitting in Deep Reinforcement Learning

By Chiyuan Zhang, Oriol Vinyals, R´emi Munos, Samy Bengio. 2018

437 citations recorded by the source corpus.

Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learning, more training power comes with a potential risk of more overfitting. As deep RL techniques are being applied to critical problems such as healthcare and finance, it is important to understand the generalization behaviors of the trained agents. In this paper, we conduct a systematic study of standard RL agents and find that they could overfit in various ways. Moreover, overfitting could happen "robustly": commonly used techniques in RL that add stochasticity do not necessarily prevent or detect overfitting. In particular, the same agents and learning algorithms could have drastically different test performance, even when all of them achieve optimal rewards during training. The observations call for more principled and careful evaluation protocols in RL. We conclude with a general discussion on overfitting in RL and a study of the generalization behaviors from the perspective of inductive bias.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, methodology): “• Side-effects (Armstrong & Levinstein, 2017; Zhang et al., 2018b; Krakovna et al., 2018): the agent is not incentivized to reduce effects unrelated to its main objectives, even if those are irreversible or difﬁcult to reverse.”
Scaling Laws for Reward Model Overoptimization (2022, background): “Overﬁtting has been observed and studied in RL settings [Zhang et al., 2018a,b, Farebrother et al., 2018, Cobbe et al., 2019].”
Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “Like [Zhang et al., 2017, Wilson et al., 2017], we consider an over-parameterized least-squares problem min x∈Rd [ f(x) := ‖Ax− y‖22 ] , where A ∈ Rn×d for d > n is the data matrix and y ∈ {−1, 1}n is the set of labels.”
On the Binding Problem in Artificial Neural Networks (2020, background): “…in gameplay (Mnih et al., 2015; Silver et al., 2017; Berner et al., 2019), it is found that agents are fragile under distributional shift (Kansky et al., 2017; Zhang et al., 2018; Gamrian and Goldberg, 2019) and require substantially more training data than humans (Tsividis et al., 2017).”
Relational Deep Reinforcement Learning (2018, background): “Unfortunately, deep RL models still face important limitations, namely, low sample efficiency and a propensity not to generalize to seemingly minor changes in the task [4, 5, 6, 7].”
Quantifying Generalization in Reinforcement Learning (2018, methodology): “Zhang et al. (2018b) conduct experiments on procedurally generated grid-world mazes, reporting many insightful conclusions on the nature of overﬁtting in RL agents.”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “While previously mentioned studies (Zhang et al., 2018c; Cobbe et al., 2019; Justesen et al., 2018; Juliani et al., 2019) reveal intriguing trends, it is hard to draw general conclusions from so few environments.”
Interference and Generalization in Temporal Difference Learning (2020, methodology): “For the generalization experiments of Section 4 we loosely follow the setup of Zhang et al. (2018a): we train RL agents in environments where the initial state is induced by a single random seed, allowing us to have proper training and test sets in the form of mutually exclusive seeds.”
A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning (2018, methodology): “Yet recent studies on generalization in supervised learning have shown that deep neural networks, with their overparameterization and massive capacity, can lead to memorization of the data, and thus poor generalization [Zhang et al., 2018, Arpit et al., 2017].”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, methodology): “Recently, the generalization performance of RL agents has been investigated by splitting training and test environments using random seeds (Zhang et al., 2018a) and distinct sets of levels in video games (Machado et al., 2018; Cobbe et al., 2019).”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “We refer the reader to (Zhang et al., 2018; Ghosh et al., 2021; Kirk et al., 2021), on which we base our discussion of generalization in RL using a Contextual MDP (CDMP) (Hallak et al., 2015).”
Invariant Causal Prediction for Block MDPs (2020, background): “In-deed, Cobbe et al. (2018); Zhang et al. (2018b) ﬁnd that agents trained using standard methods require many thousands of environments before succeeding at ‘generalizing’ to new environments.”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, background): “Consequently, we argue that this observed degradation of generalisation might contribute to the inferior generalisation properties recently attributed to many RL agents evaluated on held out test environments (Zhang et al., 2018a;b; Zhao et al., 2019).”
Observational Overfitting in Reinforcement Learning (2019, background): “Previous work (Zhang et al., 2018c) did not find such an architectural pattern for GridWorld environments, suggesting that this effect may exist primarily for observational overfitting cases.”
Increasing generality in machine learning through procedural content generation (2019, background): “While the aforementioned work [5, 11] showed that training on a larger variety of environments can lead to more general agents, it did require a large number of training levels.”
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning (2019, background): “a new task remains an open challenge [25], [26], [27], [28].”
Robust Visual Domain Randomization for Reinforcement Learning. (2019, background): “Agents trained on one domain often fail to generalize to other domains that differ only in small ways from the original domain (Sutton, 1996; Cobbe et al., 2019; Zhang et al., 2018b; Packer et al., 2018; Zhang et al., 2018a; Witty et al., 2018; Farebrother et al., 2018).”
Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation (2018, methodology): “Even small game modifications can often lead to dramatically reduced performance, leading to the suspicion that these networks learn reactions to particular situations rather than general strategies [20, 39].”