Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning

By Kimin Lee, Kibok Lee, Jinwoo Shin, Honglak Lee. 2019

198 citations recorded by the source corpus.

Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “We do not need to train many different image-to-image models to get diverse distortions (Zhang et al., 2018; Lee et al., 2020).”
Reinforcement Learning with Augmented Data (2020, methodology): “ps an image at random across the vertical axis. Rotate: Randomly samples an angle from the following set f0;90;180;270gand rotates the image accordingly. Random convolution: First introduced in [37], augments the image color by passing the input observation through a random convolutional layer. Colo”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “There have been several investigations of generalization in RL (Farebrother et al., 2018; Packer et al., 2018; Zhang et al., 2018a; Lee et al., 2019), but progress has largely proved elusive.”
A review on Deep Reinforcement Learning for Fluid Mechanics (2019, methodology): “660 Typical examples include learning a model of the environment in such a way that errors in the model do 661 not degrade the asymptotic performance [71, 72], or wrapping redundant states into equivalent classes 662 of canonical spaces [73] to increase the data efficiency; using data augmentation a”
Domain Generalization: A Survey (2021, methodology): “One can also implement the concept of domain randomization [213], which visually randomizes an environment through, e.g., computer simulators [213] or random neural networks [214].”
Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity (2021, methodology): “In response to the recent problem raised by Lee et al. (2020) that deep neural network agents often fail to generalize to unseen environments, we consider the situation where the statistics of the foreground object, such as color or shape, is unchanged, but with the corrupted (or replaced)…”
Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning (2021, methodology): “As opposed to the original jumping task ( c.f. Table 1), data augmentation inhibits generalization since RandConv forces the agent to ignore color, conﬂating the red and green tasks (Figure G.6).”
Transient Non-stationarity and Generalisation in Deep Reinforcement Learning (2020, background): “…generalisation, including to provide more diverse training environments (Tobin et al., 2017), inject noise into the environment (Stulp et al., 2011; Lee et al., 2020), incorporate inductive biases in the architecture (Kansky et al., 2017), or regularise the network (Cobbe et al., 2019b; Igl et…”
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, methodology): “ [35, 48, 10, 8, 33]. Recent results have shown that the use of transformations common in the supervised learning paradigm (such as cutout, color-jitter, or random shifts) can also be beneﬁcial in RL [11, 37, 32, 34]. However, as was noted by Kostrikov et al. [32], there are important differences be”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, methodology): “Data-augmentation for Reinforcement Learning Many image augmentation techniques such as random convolution, random shift, l2-regularization, dropout, batch normalization and noise injection are shown to improve the generalization of RL [5, 16, 22, 24, 33, 50].”
Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “Following [48], we use He initialization [34] for the weights and fixed zero bias, sampling the size of the kernel uniformly across odd values ranging from 1 to 11.”
Neuroevolution of self-interpretable agents (2020, methodology): “ies of objects of interest, where our method may perform as poorly (or worse) than methods that do not require sparse attention in pixel space. We believe this work complements other approaches (e.g. [3, 44, 45, 51, 57, 69, 86, 101, 105]) that approach the generalization problem, and future work wil”
Automatic Data Augmentation for Generalization in Reinforcement Learning (2021, methodology): “, 2019), data augmentation (Cobbe et al., 2018; Lee et al., 2020; Kostrikov et al., 2020; Laskin et al., 2020), or representation learning (Igl et al.”