Parameter-exploring policy gradients

By Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, J. Urgen Schmidhuber. 2010

301 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Molecular Biology, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Molecular Biology

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Deep learning in neural networks: An overview (2014, neutral): “…et al., 2010;Heess, Silver, & Teh, 2012; Kohl & Stone, 2004; Peters, 2010; Peters & Schaal, 2008a, 2008b; Rückstieß, Felder, & Schmidhuber, 2008; Sehnke et al., 2010; Sutton, McAllester, Singh, &Mansour, 1999;Wierstra, Foerster, Peters, & Schmidhuber, 2010; Wierstra, Schaul, Peters, &…”
World Models (2018, methodology): “, 2013), evolution-based algorithms have been shown to be effective at solving RL tasks (Stanley & Miikkulainen, 2002; Gomez et al., 2008; Gomez & Schmidhuber, 2005; Gauci & Stanley, 2010; Sehnke et al., 2010; Miikkulainen, 2013).”
Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017, background): “…case where p ψ is factored Gaussian (as in this work), the resulting gradient estimator is also known as simultaneous perturbation stochastic approximation [Spall, 1992], parameter-exploring policy gradients [Sehnke et al., 2010], or zero-order gradient estimation [Nesterov and Spokoiny, 2011].”
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning (2017, methodology): “…of deep learning algorithms have shown promise on RL problems so far: Q-learning methods such as DQN (Mnih et al., 2015), policy gradient methods (Sehnke et al., 2010) (e.g. A3C (Mnih et al., 2016), TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017)), and more recently evolution…”
Parameter Space Noise for Exploration (2017, background): “Their main limitation, however, is that they are either only proposed and evaluated for the on-policy setting with relatively small and shallow function approximators [10] or disregard all temporal structure and gradient information [9, 11, 12].”
One Big Net For Everything (2018, background): “6.6], or reinforcement learning [29, 90, 101, 76] through policy gradients [104, 91, 3, 1, 13, 30, 103, 48, 82, 22, 102, 42] [76, Sec.”
Evolved Policy Gradients (2018, neutral): “Similar formulations also appear in prior works including (36; 32; 16).”
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement (2019, methodology): “A parameter exploring policy gradients (Sehnke et al., 2010) based black-box optimization, which is similar to the one used in (Zhang et al., 2018), is also compared.”
A Survey on Policy Search for Robotics (2013, methodology): “Such an approach was first introduced by [74, 75] with the Parameter Exploring Policy Gradient (PEPG) algorithm.”
On Ensuring that Intelligent Machines Are Well-Behaved (2017, methodology): “We will take a nonstandard—but not uncommon [Kober and Peters, 2009; Sehnke et al., 2010; Theodorou et al., 2010; Stulp and Sigaud, 2012]—approach to reinforcement learning: we propose an algorithm that searches the space of distributions over policies rather than an algorithm that, like SPI,…”
Intrinsically motivated information foraging (2010, methodology): “direct search methods: a policy gradient method, PGPE (policy gradients with parameter exploration, [19]), and an evolutionary algorithm, CMA-ES (covariance matrix adaptation evolutionary strategy, [20]).”
Randomized Value Functions via Multiplicative Normalizing Flows (2018, background): “This is sometimes called state-dependent exploration (Sehnke et al., 2010) as the same action will be taken every time the same state is sampled in the episode.”
Model-Based Policy Gradients with Parameter-Based Exploration by Least-Squares Conditional Density Estimation (2013, methodology): “The RL methods developed so far can be categorized into two types: Policy iteration where policies are learned based on value function approximation [21, 12] and policy search where policies are learned directly to maximize expected future rewards [24, 4, 22, 8, 15, 26].”