Noisy Networks for Exploration

By Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, R´emi Munos, Demis Hassabis, Olivier Pietquin and 2 more. 2017

1,022 citations recorded by the source corpus.

We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and dueling agents (entropy reward and $\epsilon$-greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, background): “…with double Q-learning [van Hasselt, 2010, van Hasselt et al., 2016], dueling network architectures [Wang et al., 2016], prioritised experience replay [Schaul et al., 2016], noisy networks for exploration [Fortunato et al., 2017], and distributional reinforcement learning [Bellemare et al., 2017].”
Surprising Negative Results for Generative Adversarial Tree Search (2018, background): “[12] propose a surrogate to optimism, [17] suggest noisy exploration.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Noisy Nets (Fortunato et al. 2017) propose a noisy linear layer that combines a deterministic and noisy stream, where b and w are random variables, and (cid:12) denotes the element-wise product.”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “lel for the Ape-X DQN and share a prioritized experience replay memory [50]. DQN was extended to double DQN (DDQN) [121, 122] which helps exploration as the overestimation bias is reduced. Noisy DQNs [20] explore by a stochastic layer in the policy network (see [40, 101]). Distributional Q-learning ”
Meta-Gradient Reinforcement Learning (2018, background): “, 2017], and parameter noise for exploration [Fortunato et al., 2018].”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Noise in Parameter Space for Exploration (Fortunato et al., 2018; Plappert et al., 2018): Pertubates network parameters to aid exploration.”
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “For comparison, Deep Q-Networks trained on Atari games were updated after every 4 actions [29, 3, 20, 30].”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “In the pursuit of developing efﬁcient and stable off-policy deep RL algorithms, recent papers have presented various conceptual and engineering ideas ( e.g., [41, 48, 59, 60, 45, 19, 1, 4, 13, 22]).”
Exploration by Random Network Distillation (2018, background): “, 2017), parameter noise (Plappert et al., 2017; Fortunato et al., 2017), identifying diverse policies (Eysenbach et al.”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “, 2016), Noisy Networks (Fortunato et al., 2017), and multi-step updates (Sutton, 1988).”
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning (2017, methodology): “, 2017) algorithm that combines many different recent improvements to DQN (Van Hasselt et al., 2016; Wang et al., 2015; Schaul et al., 2015; Sutton & Barto, 1998; Bellemare et al., 2017; Fortunato et al., 2017).”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “to deep reinforcement learning, each attaining significant outperformance over deep RL baselines on certain tasks [20, 47, 35, 11, 17].”
Parameter Space Noise for Exploration (2017, methodology): “Concurrently to our work, [45] have proposed a similar approach that utilizes parameter perturbations for more efficient exploration.”
Large-Scale Study of Curiosity-Driven Learning (2018, methodology): “Other methods of exploration are designed to work in combination with maximizing a reward function, such as those utilizing uncertainty about value function estimates [5, 23], or those using perturbations of the policy for exploration [8, 29].”
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches (2018, methodology): “Weight noise was found to aid exploration in reinforcement learning (Plappert et al., 2017; Fortunato et al., 2017).”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, methodology): “Furthermore, we compared against scores of Bootstrap DQN (Osband et al., 2016), NoisyNet (Fortunato et al., 2017), CTS, Pixel, Re-actor (Ostrovski et al., 2017) which are borrowed from the original papers.”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “…our algorithm is related to recent approaches in deep learning for exploration to avoid local minima, e.g., natural evolution strategy (Wierstra et al., 2014), entropy-SGD (Chaudhari et al., 2016), and noisy networks for reinforcement learning (Fortunato et al., 2018; Plappert et al., 2018).”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “Since then, multiple improvements have been proposed such as Double Q Learning (Van Hasselt et al., 2016), Dueling Network Architectures (Wang et al., 2015), Prioritized Experience Replay (Schaul et al., 2015), and Noisy Networks (Fortunato et al., 2017).”
AI Safety Gridworlds (2017, background): “Finally, agents based on deep architectures could benefit from the incorporation of better uncertainty estimates in neural networks (Gal, 2016; Fortunato et al., 2017).”