Generalization and Regularization in DQN

By Jesse Farebrother, Marlos C. Machado, Michael H. Bowling. 2018

227 citations recorded by the source corpus.

Deep reinforcement learning (RL) algorithms have shown an impressive ability to learn complex control policies in high-dimensional environments. However, despite the ever-increasing performance on popular benchmarks such as the Arcade Learning Environment (ALE), policies learned by deep RL algorithms often struggle to generalize when evaluated in remarkably similar environments. In this paper, we assess the generalization capabilities of DQN, one of the most traditional deep RL algorithms in the field. We provide evidence suggesting that DQN overspecializes to the training environment. We comprehensively evaluate the impact of traditional regularization methods, $\ell_2$-regularization and dropout, and of reusing the learned representations to improve the generalization capabilities of DQN. We perform this study using different game modes of Atari 2600 games, a recently introduced modification for the ALE which supports slight variations of the Atari 2600 games traditionally used for benchmarking. Despite regularization being largely underutilized in deep RL, we show that it can, in fact, help DQN learn more general features. These features can then be reused and fine-tuned on similar tasks, considerably improving the sample efficiency of DQN.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, background): “Overﬁtting has been observed and studied in RL settings [Zhang et al., 2018a,b, Farebrother et al., 2018, Cobbe et al., 2019].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Generalization in RL has been investigated by creating different game modes (Farebrother, Machado, and Bowling, 2018) and game levels (Nichol, Pfau, Hesse, Klimov, and Schulman, 2018) for training and testing, by exposing internal parameters of various classic environments (Packer, Gao, Kos,…”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “using dropout (Farebrother et al., 2018) and cutout (Cobbe et al.”
Quantifying Generalization in Reinforcement Learning (2018, background): “(Farebrother et al., 2018) also address this problem, accurately recognizing that conflating train and test environments has contributed to the lack of regularization in deep RL.”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “There have been several investigations of generalization in RL (Farebrother et al., 2018; Packer et al., 2018; Zhang et al., 2018a; Lee et al., 2019), but progress has largely proved elusive.”
Interference and Generalization in Temporal Difference Learning (2020, background): “This led to many works showing the brittleness of new TD methods (Machado et al., 2018; Farebrother et al., 2018; Packer et al., 2018; Witty et al., 2018), and works proposing to train on a distribution of environments (Zhang et al.”
Emergent Social Learning via Multi-agent Reinforcement Learning (2020, unknown): “Since deep RL often generalizes poorly (Cobbe et al., 2019; Farebrother et al., 2018; Packer et al., 2018), we believe these results are a promising first step on a new path for enhancing generalization.”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, methodology): “Fare-brother et al. (2018) and Cobbe et al. (2019) showed that regularization methods can improve the generalization performance of RL agents using various game modes of Atari (Machado et al., 2018) and procedurally generated arcade environments called CoinRun, respectively.”
Automatic Data Augmentation for Generalization in Reinforcement Learning (2021, background): “Current methods fail to generalize to unseen environments even when trained on similar settings [19, 51, 71, 11, 21, 12, 60].”
Regularization Matters in Policy Optimization (2019, methodology): “The generalization gap can come from different modes/levels/difficulties of a game (Farebrother et al., 2018), simulation vs. real world (Tobin et al., 2017), parameter variations (Pattanaik et al., 2018), or different random seeds in environment generation (Zhang et al., 2018b).”
Domain Generalization with MixStyle (2021, background): “A natural way to improve generalization, which has been shown effective in (Cobbe et al., 2019; Farebrother et al., 2018), is to use regularization, e.g., weight decay.”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, methodology): “…; with the community highlighting that deep RL agents tend to overﬁt to the environments they are trained on (Cobbe et al., 2019; Witty et al., 2018; Farebrother et al., 2018; Juliani et al., 2019; Kostrikov et al., 2020; Song et al., 2019; Justesen et al., 2018; Packer et al., 2018).”
Robust Visual Domain Randomization for Reinforcement Learning. (2019, background): “Agents trained on one domain often fail to generalize to other domains that differ only in small ways from the original domain (Sutton, 1996; Cobbe et al., 2019; Zhang et al., 2018b; Packer et al., 2018; Zhang et al., 2018a; Witty et al., 2018; Farebrother et al., 2018).”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “We note that different Atari game modes may also be used to produce variation and assess generalization capability, as proposed by Farebrother et al. (2018).”
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, methodology): “Farebrother et al. [17] and Cobbe et al. [11] show that such regularization methods can improve the generalization ability of RL agents in Atari [39] and CoinRun [11], respectively.”
Active Domain Randomization (2019, background): “Generalization in RL has long been one of the holy grails of the ﬁeld, and recent work like Packer et al. (2018), Cobbe et al. (2018), and Farebrother et al. (2018) highlight the tendency of deep RL policies to overﬁt to details of the training environment.”