Parameter Space Noise for Exploration

By Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, Marcin Andrychowicz. 2017

689 citations recorded by the source corpus.

Deep reinforcement learning (RL) methods generally engage in exploratory behavior through noise injection in the action space. An alternative is to add noise directly to the agent's parameters, which can lead to more consistent exploration and a richer set of behaviors. Methods such as evolutionary strategies use parameter perturbations, but discard all temporal structure in the process and require significantly more samples. Combining parameter noise with traditional RL methods allows to combine the best of both worlds. We demonstrate that both off- and on-policy methods benefit from this approach through experimental comparison of DQN, DDPG, and TRPO on high-dimensional discrete action environments as well as continuous control tasks. Our results show that RL with parameter noise learns more efficiently than traditional RL with action space noise and evolutionary strategies individually.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, methodology): “First, we change evaluation in DDPG (Plappert et al. 2017) such that during evaluation at the end of an epoch, 10 full trajectories are evaluated.”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Our learned action space is agnostic to the control algorithm and hence other exploration algorithms [24, 4, 26] can also benefit from using it.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Noise in Parameter Space for Exploration (Fortunato et al., 2018; Plappert et al., 2018): Pertubates network parameters to aid exploration.”
Uncertainty-based Continual Learning with Adaptive Regularization (2019, background): “Moreover, a possible reason why UCL achieves such high rewards in RL setting may be due to the by-product of our weight sampling procedure; namely, the Gaussian perturbation of the weights for the variational inference enables an effective exploration of policies for RL as suggested in [21].”
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling (2018, methodology): “Direct Noise Injection Parameter-Noise [20] is a recently proposed approach for exploration in deep RL that has shown promising results.”
Exploration by Random Network Distillation (2018, methodology): “Other methods of exploration include adversarial self-play (Sukhbaatar et al., 2018), maximizing empowerment (Gregor et al., 2017), parameter noise (Plappert et al., 2017; Fortunato et al., 2017), identifying diverse policies (Eysenbach et al., 2018; Achiam et al., 2018), and using ensembles of…”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, background): “The problem of exploration can be addressed by using parameters noise for exploration (Plappert et al., 2017) and faster information propagation could be achieved by employing multi-step returns.”
Noisy Networks for Exploration (2017, methodology): “[27] Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz.”
Randomized Prior Functions for Deep Reinforcement Learning (2018, background): “The action mask W means this cannot be solved easily by evolution or policy search evolution, unlike previous ‘chain’ examples [47, 54].”
Simple random search provides a competitive approach to reinforcement learning (2018, methodology): “Our ﬁndings contradict the common belief that policy gradient techniques, which rely on exploration in the action space, are more sample eﬃcient than methods based on ﬁnite-diﬀerences [25, 26].”
Large-Scale Study of Curiosity-Driven Learning (2018, methodology): “Other methods of exploration are designed to work in combination with maximizing a reward function, such as those utilizing uncertainty about value function estimates [5, 23], or those using perturbations of the policy for exploration [8, 29].”
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning (2018, positive): “For exploration, we use parameter-space noise (Plappert et al., 2017).”
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches (2018, methodology): “Weight noise was found to aid exploration in reinforcement learning (Plappert et al., 2017; Fortunato et al., 2017).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “…our algorithm is related to recent approaches in deep learning for exploration to avoid local minima, e.g., natural evolution strategy (Wierstra et al., 2014), entropy-SGD (Chaudhari et al., 2016), and noisy networks for reinforcement learning (Fortunato et al., 2018; Plappert et al., 2018).”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “The stochastic perturbation is performed because IM agents have two layers of exploration mechanisms: (1) the higher-level IR incentive that rewards when new states are reached, and (2) a more basic exploratory mechanism such as epsilon-greedy exploration, action-space noise, or parameter-space nois”
Learning Awareness Models (2018, methodology): “A sea of other methods have been proposed for exploration (MacKay, 1992; Ghavamzadeh et al., 2015; Asmuth et al., 2009; Gal, 2016; Stachniss et al., 2005; Plappert et al., 2017; Fu et al., 2017).”
Meta-Reinforcement Learning of Structured Exploration Strategies (2018, background): “…& Littman, 2008; Bellemare et al., 2016), Thompson sampling and boot-strapped models (Chapelle & Li, 2011; Osband et al., 2016), optimism in the face of uncertainty (Brafman & Tennen-holtz, 2003; Kearns & Singh, 2002), and parameter space exploration (Plappert et al., 2017; Fortunato et al., 2017).”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Parameter-space exploration [45, 46] methods do not explore at each time step, but rather generate correlated behaviors based on explored parameters at the start.”