Recurrent policy gradients

By Daan Wierstra, Alexander Förster, Jan Peters, J. Urgen Schmidhuber. 2010

134 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Gradient Estimation Using Stochastic Computation Graphs (2015, methodology): “Appendix C recalls several classic and recent techniques in variational inference [14, 10, 21] and reinforcement learning [23, 25, 15], where the loss functions can be straightforwardly described using 1”
Deep learning in neural networks: An overview (2014, methodology): “An important class of DS methods for NNs are Policy Gradient methods (Williams, 1986, 1988, 1992a; Baxter and Bartlett, 1999; Sutton et al., 1999a; Aberdeen, 2003; Ghavamzadeh and Mahadevan, 2003; Kohl and Stone, 2004; Wierstra et al., 2007, 2008; Rückstieß et al., 2008; Peters and Schaal, 2008b,a; ”
Learning to Compose Domain-Specific Transformations for Data Augmentation (2017, positive): “We compute the recurrent policy gradient [32]”
Cognitive Mapping and Planning for Visual Navigation (2017, positive): “Follow-up works (Mnih et al. 2016; Gu et al. 2016; Schulman et al. 2015) propose improvements to DRL algorithms, (Oh et al. 2016; Mnih et al. 2016; Wierstra et al. 2010; Heess et al. 2015; Zhang et al. 2016) study how to incorporate memory into such neural network based models.”
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, background): “lly it is to be noted that although actions are stochastically sampled from the outputs at each time step, the hidden states that are passed on serve as a sufﬁcient statistic for x 0;a 0:::x t 1;a t 1Wierstra et al. (2010). 3.2.2 LAYER SHRINKAGE The length of the trajectory for layer shrinkage is T ”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “1, except that C is now an FNN or RNN trained by evolutionary algorithms [200, 255, 105, 56, 68] applied to NNs [165, 321, 180, 259, 72, 90, 89, 110, 94], or by policy gradient methods [314, 315, 316, 274, 18, 1, 63, 128, 313, 210, 192, 191, 256, 85, 312, 190, 82, 93][245, Sec.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “Note that RNNs are not restricted to value-function-based methods but have also been successfully applied to policy search [138] and actor-critic methods [36, 72].”
Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning (2018, background): “In partially observable settings, the networks are typically recurrent, e.g., LSTMs (Wierstra et al., 2009), as they can learn to represent a sufﬁcient statistic of the action-observation history τ at in the hidden activations.”
One Big Net For Everything (2018, background): “6.6], or reinforcement learning [29, 90, 101, 76] through policy gradients [104, 91, 3, 1, 13, 30, 103, 48, 82, 22, 102, 42] [76, Sec.”
Self-Delimiting Neural Networks (2012, background): “…through a focus on reducing search spaces by co-evolving the comparatively small weight vectors of individual neurons and synapses [19], by Natural Gradient-based Stochastic Search Strategies [80, 73, 74, 48, 17, 79], and by reducing search spaces through weight matrix compression [55, 35].”
Imitating driver behavior with generative adversarial networks (2017, background): “policies [30], [31] disambiguate perceptually similar states by acting with respect to histories of, rather than individual, observations.”
Options Discovery with Budgeted Reinforcement Learning (2016, methodology): “…trajectories used for approximating the gradient using Monte Carlo sampling techniques, b t is a variance reduction term at time t estimated during learning, and we consider that future actions do not depend on past rewards (see [ Wierstra et al. , 2010 ] for details on recurrent policy gradients).”
Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks (2019, background): “Such memory-based policies have been primarily studied in the context of robot navigation in partially observable environments, where the neural network learns to encode the received observations and write them into a map-like memory [15, 16, 18, 32, 43].”
Time Limits in Reinforcement Learning (2017, background): “In deep learning (LeCun et al., 2015; Schmidhuber, 2015), it is highly common to use a stack of previous observations or recurrent neural networks (RNNs) (Goodfellow et al., 2016) to address partial observations (Wierstra et al., 2009).”
Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon (2018, background): “This can be applied in the imitation learning setting, as well as in RL, for instance by learning a recurrent policy (Wierstra et al., 2010).”
Reward-based training of recurrent neural networks for cognitive and value-based tasks (2016, methodology): “In practice, however, it is more useful to have a history-dependent baseline that attempts to predict the future return at every time (Wierstra et al., 2009; Zaremba & Sutskever, 2016).”
Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning (2017, background): “The baseline function can be calculated in diﬀerent manners [32, 40].”