Observational Overfitting in Reinforcement Learning

By Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, Behnam Neyshabur. 2019

150 citations recorded by the source corpus.

A major component of overfitting in model-free reinforcement learning (RL) involves the case where the agent may mistakenly correlate reward with certain spurious features from the observations generated by the Markov Decision Process (MDP). We provide a general framework for analyzing this scenario, which we use to design multiple synthetic benchmarks from only modifying the observation space of an MDP. When an agent overfits to different observation spaces even if the underlying MDP dynamics is fixed, we term this observational overfitting. Our experiments expose intriguing properties especially with regards to implicit regularization, and also corroborate results from previous works in RL generalization and supervised learning (SL).

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, background): “A series of recent studies show that RL agents fail to generalize to new environments, even when they are trained on a large set of diverse yet semantically similar ones [17, 19, 59, 11, 12, 50].”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “Overfitting and lack of generalization ability have been raised as the most critical problems of deep reinforcement learning (RL) [5, 8, 30, 34, 38, 47, 52].”
The alignment problem from a deep learning perspective (2022, background): “The examples of goal misgeneralization discussed in Section 3.1 were caused by spurious correlations between rewards and environmental features on small-scale tasks (also known as “observational overﬁtting”) [Song et al., 2019].”
Fast Adaptation to New Environments via Policy-Dynamics Value Functions (2020, background): “However, recent studies have pointed out that RL agents trained and tested on the same environment tend to overfit to that environment’s idiosyncracies and are unable to generalize to even small perturbations (Whiteson et al., 2011; Rajeswaran et al., 2017; Zhang et al., 2018c;a; Henderson et al., 2”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, methodology): “In this section, we exhibit that PSM ignores spurious information for generalization using a LQR task (Song et al., 2019) with non-image inputs.”
Invariant Causal Prediction for Block MDPs (2020, background): “In the worst case, some training environments may contain spurious correlations that will not be present at test time, causing catastrophic failures in generalization (Zhang et al., 2018a; Song et al., 2020).”
Neuroevolution of self-interpretable agents (2020, background): “[3, 32, 45, 46, 52, 58, 70, 87, 102, 106]) that approach the generalization problem, and future work will continue to develop on these ideas to push the generalization abilities proposed in more general domains [7, 16, 17, 49, 59, 70, 105].”
Automatic Data Augmentation for Generalization in Reinforcement Learning (2021, background): “A series of recent studies show that RL agents fail to generalize to new environments (Farebrother et al., 2018; Gamrian & Goldberg, 2019; Zhang et al., 2018; Cobbe et al., 2018; 2019; Song et al., 2020).”