State-Dependent Exploration for Policy Gradient Methods

By Thomas Rückstieß, Martin Felder, J. Urgen Schmidhuber. 2008

101 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, background): “Such unstructured exploration causes a large variance which grows with the number of time steps, and perturbs actions too frequently [35, 18].”
Reinforcement learning in robotics: A survey (2013, background): “Alternatively, one could generate a form of structured, state-dependent exploration (Rückstieß et al., 2008) ǫ(μ(s, t)) = εt μ(s, t) with [εt]ij ∼ N (0, σ 2 ij), where σ2 ij are meta-parameters of the exploration that can also be optimized.”
Hierarchical Variational Models (2015, methodology): “A similar methodology also arises in the policy search literature (Rückstieß et al., 2008; Sehnke et al., 2008).”
Parameter Space Noise for Exploration (2017, background): “Their main limitation, however, is that they are either only proposed and evaluated for the on-policy setting with relatively small and shallow function approximators [10] or disregard all temporal structure and gradient information [9, 11, 12].”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “1, except that C is now an FNN or RNN trained by evolutionary algorithms [200, 255, 105, 56, 68] applied to NNs [165, 321, 180, 259, 72, 90, 89, 110, 94], or by policy gradient methods [314, 315, 316, 274, 18, 1, 63, 128, 313, 210, 192, 191, 256, 85, 312, 190, 82, 93][245, Sec.”
One Big Net For Everything (2018, background): “6.6], or reinforcement learning [29, 90, 101, 76] through policy gradients [104, 91, 3, 1, 13, 30, 103, 48, 82, 22, 102, 42] [76, Sec.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “The stochastic perturbation is performed because IM agents have two layers of exploration mechanisms: (1) the higher-level IR incentive that rewards when new states are reached, and (2) a more basic exploratory mechanism such as epsilon-greedy exploration, action-space noise, or parameter-space nois”
A Survey on Policy Search for Robotics (2013, methodology): “We categorize model-free policy search approaches based on their policy evaluation strategies, their policy update strategies [58, 59], and their exploration strategies [39, 68].”
Learning model-free robot control by a Monte Carlo EM algorithm (2009, methodology): “We adopt here the same controller parametrization that was used by Kober and Peters (2009) based on a model suggested by Rückstieß et al. (2008), in which the controller is linear-Gaussian with state dependent noise: u t = (θ + ε t ) φ(x t ), (17) where φ : R n → R d are fixed basis functions and ε…”
Learning motor primitives for robotics (2009, background): “Previous work in [6], [16], with the notable exception of [19], has focused on state-independent, white Gaussian exploration, i.”
Exploring parameter space in reinforcement learning (2010, methodology): “An alternative to parameter-based exploration, that addresses most of the shortcomings of action-based exploration is State-Dependent Exploration [18].”
Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (2008, methodology): “Other recent RL techniques for RNN are based on the concept of olicy gradients [110, 119, 118, 56, 100, 117].”
Parameter-exploring policy gradients (2010, unknown): “A related approach is taken in Rückstieß et al. (2008) where the exploratory noise in each time step is replaced by a state-dependent exploratory function (SDE) with additional parameters.”
Exploration in Deep Reinforcement Learning: A Survey (2022, background): “In [157], the problem of assigning rewards when the same state is present multiple times is discussed.”