Automatic Data Augmentation for Generalization in Deep Reinforcement Learning

By Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, Rob Fergus. 2020

120 citations recorded by the source corpus.

Deep reinforcement learning (RL) agents often fail to generalize to unseen scenarios, even when they are trained on many instances of semantically similar environments. Data augmentation has recently been shown to improve the sample efficiency and generalization of RL agents. However, different tasks tend to benefit from different kinds of data augmentation. In this paper, we compare three approaches for automatically finding an appropriate augmentation. These are combined with two novel regularization terms for the policy and value function, required to make the use of data augmentation theoretically sound for certain actor-critic algorithms. We evaluate our methods on the Procgen benchmark which consists of 16 procedurally-generated environments and show that it improves test performance by ~40% relative to standard RL algorithms. Our agent outperforms other baselines specifically designed to improve generalization in RL. In addition, we show that our agent learns policies and representations that are more robust to changes in the environment that do not affect the agent, such as the background. Our implementation is available at this https URL.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “Data augmentation techniques are widely used in computer vision tasks [21, 40] and have recently been applied to deep RL tasks [18, 20, 33, 26].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “Multiple follow ups to our initial preprint appeared on ArXiv (Raileanu et al., 2020; Okada & Taniguchi, 2020), using similar techniques on other tasks, thus supporting the effectiveness and generality of data augmentation in RL.”
Prioritized Level Replay (2020, background): “, 2019), data augmentation (Kostrikov et al., 2020; Raileanu et al., 2020), and knowledge distillation (Igl et al.”
Tell me why! - Explanations support learning of relational and causal structure (2021, methodology): “complicated techniques, for example data augmentation (Raileanu et al., 2020; Guan et al., 2021), or auxiliary generative model learning (Gregor et al.”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “The use of such data augmentations now forms an essential component of several recent visual RL algorithms [Srinivas et al., 2020, Raileanu et al., 2020, Yarats et al., 2021a, Stooke et al., 2020, Hansen and Wang, 2021, Schwarzer et al., 2020b].”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, background): “PSEs are complementary to data augmentation methods (Kostrikov et al., 2020; Lee et al., 2020a; Raileanu et al., 2020; Ye et al., 2020), which have recently been shown to signiﬁcantly improve agents’ generalization capabilities.”
Tell me why! Explanations support learning relational and causal structure (2021, background): “Most of our tasks could potentially be learned with sufﬁcient data alone, especially if combined with more complicated techniques, for example data augmentation (Raileanu et al., 2020; Guan et al., 2021), or auxiliary generative model learning (Gregor et al., 2019).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “[81] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus.”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, methodology): “Data-augmentation for Reinforcement Learning Many image augmentation techniques such as random convolution, random shift, l2-regularization, dropout, batch normalization and noise injection are shown to improve the generalization of RL [5, 16, 22, 24, 33, 50].”