Assessing Generalization in Deep Reinforcement Learning

By Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krahenbuhl, Vladlen Koltun, Dawn Song. 2018

279 citations recorded by the source corpus.

Deep reinforcement learning (RL) has achieved breakthrough results on many tasks, but agents often fail to generalize beyond the environment they were trained in. As a result, deep RL algorithms that promote generalization are receiving increasing attention. However, works in this area use a wide variety of tasks and experimental setups for evaluation. The literature lacks a controlled assessment of the merits of different generalization schemes. Our aim is to catalyze community-wide progress on generalization in deep RL. To this end, we present a benchmark and experimental protocol, and conduct a systematic empirical study. Our framework contains a diverse set of environments, our methodology covers both in-distribution and out-of-distribution generalization, and our evaluation includes deep RL algorithms that specifically tackle generalization. Our key finding is that `vanilla' deep RL algorithms generalize better than specialized schemes that were proposed specifically to tackle generalization.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “…of different environments were used in experiments in the generalization for RL literature, which have played an important role in advancing the ﬁeld (Packer et al., 2018; Rajeswaran et al., 2017; Pinto et al., 2017; Yu et al., 2017; Henderson et al., 2017a; Dulac-Arnold et al., 2020;…”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2018) for training and testing, by exposing internal parameters of various classic environments (Packer et al., 2018), and by procedurally generating environments (Justesen et al.”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, methodology): “, 2017] and were modified by adapting code from [Packer et al., 2018].”
Quantifying Generalization in Reinforcement Learning (2018, background): “Packer et al. (2018) propose a diﬀerent benchmark to measure generalization using six classic environments, each of which has been modiﬁed to expose several internal parameters.”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “There have been several investigations of generalization in RL (Farebrother et al., 2018; Packer et al., 2018; Zhang et al., 2018a; Lee et al., 2019), but progress has largely proved elusive.”
Meta-Learning in Neural Networks: A Survey (2020, neutral): “Several multi-MDP benchmarks [239], [240] have”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “This complicates the task of evaluating and comparing learning algorithms but is necessary and we believe, feasible, e.g. see Yu et al. (2017); Packer et al. (2018); Chevalier-Boisvert et al. (2018); Dulac-Arnold et al. (2020); Ahmed et al. (2020).”
Active Domain Randomization (2019, background): “Generalization in RL has long been one of the holy grails of the ﬁeld, and recent work like Packer et al. (2018), Cobbe et al. (2018), and Farebrother et al. (2018) highlight the tendency of deep RL policies to overﬁt to details of the training environment.”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, methodology): “[Packer et al., 2018] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song.”
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, background): “Generalization in RL can be split into different categories, such as generalization across different visual inputs (with the same semantics) [11, 19, 37], dynamics [42], or other environment structures [3, 55, 12].”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, background): “…; with the community highlighting that deep RL agents tend to overﬁt to the environments they are trained on (Cobbe et al., 2019; Witty et al., 2018; Farebrother et al., 2018; Juliani et al., 2019; Kostrikov et al., 2020; Song et al., 2019; Justesen et al., 2018; Packer et al., 2018).”
Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning (2019, methodology): “The generalization of RL agents can be characterized by visual changes (Cobbe et al., 2019; Gam-rian & Goldberg, 2019), different dynamics (Packer et al., 2018), and various structures (Beattie et al., 2016; Wang et al., 2016).”
Learning Domain Invariant Representations in Goal-conditioned Block MDPs (2021, background): “They also tend to overfit to specific environments and often fail to generalize beyond the environment they were trained on [4].”
Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation (2020, unknown): “In the context of domain-shift between training and testing, [29] and [14] further highlighted the problem that deep RL agents often fail to generalise, e.”
Generalization to New Actions in Reinforcement Learning (2020, unknown): “Previously, state-space generalization has been used to transfer policies to new environments (Cobbe et al., 2018; Nichol et al., 2018; Packer et al., 2018), agent morphologies (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), and visual inputs for manipulation of unseen…”
Robust Visual Domain Randomization for Reinforcement Learning. (2019, background): “Agents trained on one domain often fail to generalize to other domains that differ only in small ways from the original domain (Sutton, 1996; Cobbe et al., 2019; Zhang et al., 2018b; Packer et al., 2018; Zhang et al., 2018a; Witty et al., 2018; Farebrother et al., 2018).”
Neuroevolution of self-interpretable agents (2020, background): “[3, 32, 45, 46, 52, 58, 70, 87, 102, 106]) that approach the generalization problem, and future work will continue to develop on these ideas to push the generalization abilities proposed in more general domains [7, 16, 17, 49, 59, 70, 105].”
Regularization Matters in Policy Optimization (2019, background): “There are also some comprehensive studies on RL generalization with interesting findings (Zhang et al., 2018a;b; Zhao et al., 2019; Packer et al., 2018), e.g., algorithms performing better in training environment could perform worse with domain shift (Zhao et al., 2019).”
Interference and Generalization in Temporal Difference Learning (2020, background): “This led to many works showing the brittleness of new TD methods (Machado et al., 2018; Farebrother et al., 2018; Packer et al., 2018; Witty et al., 2018), and works proposing to train on a distribution of environments (Zhang et al.”
The Measure of Intelligence (2019, unknown): “[50, 70, 17, 49]), which was previously a stark problem [76, 35, 101, 70], as RL agents have for a long time been tested on their training data.”