Solving Deep Memory POMDPs with Recurrent Policy Gradients

By Daan Wierstra, Alexander Förster, Jan Peters, J. Urgen Schmidhuber. 2007

201 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Reinforcement Learning

Citations in the corpus

Benchmarking Deep Reinforcement Learning for Continuous Control (2016, methodology): “Similar tasks have been explored in Gomez & Miikkulainen (1998); Sch¨afer & Udluft (2005); Heess et al. (2015a); Wierstra et al. (2007).”
Memory-based control with recurrent neural networks (2015, methodology): “[30] trained LSTM-based stochastic policies using Reinforce; [31, 29, 32] used actor-critic architectures.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “Using this approach, the decoder maintains an internal memory of the agent’s entire preceeding history, which is essential for navigating in a partially observable environment [47].”
Recurrent Models of Visual Attention (2014, methodology): “But this is just the gradient of the RNN that defines our agent evaluated at time step t and can be computed by standard backpropagation [25].”
Deep learning in neural networks: An overview (2014, methodology): “olicy Gradient methods (Williams, 1986, 1988, 1992a; Baxter and Bartlett, 1999; Sutton et al., 1999a; Baxter and Bartlett, 2001; Aberdeen, 2003; Ghavamzadeh and Mahadevan, 2003; Kohl and Stone, 2004; Wierstra et al., 2007, 2008; Ruckstieß et al., 2008; Peters and¨ Schaal, 2008b,a; Sehnke et al., 201”
Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation (2018, methodology): “,ot ) when using policy gradient [39] and Q-learning [14].”
Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability (2017, background): “As Recurrent Neural Networks (RNNs) inherently maintain an internal state ht to compress input history until timestep t, they have been demonstrated to be effective for learning POMDP policies (Wierstra et al., 2007).”
Deep Recurrent Q-Learning for Partially Observable MDPs (2015, methodology): “Previously, LSTM networks have been demonstrated to solve POMDPs when trained using policy gradient methods (Wierstra et al. 2007).”
Deep Sequential Neural Network (2014, methodology): “More precisely, our learning method is close to the methods proposed in [18] and [12] with the difference that, instead of considering a reward signal which is usual in reinforcement learning, we consider a loss function ∆ computing the quality of the system.”
Deep Variational Reinforcement Learning for POMDPs (2018, methodology): “There are several approaches that utilise RNN s in POMDP s (Bakker, 2002; Wierstra et al., 2007; Zhang et al., 2015; Heess et al., 2015), including multi-agent settings (Foerster et al., 2016), learning text-based fantasy games (Narasimhan et al., 2015) or, most recently, applied to Atari…”
Deep Sequential Neural Networks (2014, methodology): “Instead of using this estimate, we replace ∆(F (xi, H), y) by ∆(F (xi, H), y) − b where b = Ep(x,H,y)[∆(F (xi, H), y)] which can be easily estimated on the training set [18].”
Evolving Memory Cell Structures for Sequence Learning (2009, background): “recurrent policy gradient algorithm [18], a learning rate of 0.”
Graph Classification using Structural Attention (2018, methodology): “To compute ∇θ logπ (rt [τ (ci t+1)]|s 1:t ;θ ), we simply compute the gradient of our network at each time step, this can be done using standard backpropagation [31].”
Accelerated Neural Evolution through Cooperatively Coevolved Synapses (2008, methodology): “Recurrent Policy Gradients (RPG; Wierstra et al., 2007) where a stochastic policy is represented by a Long Short-Term Memory network (LSTM; Hochreiter and Schmidhuber, 1997) trained with BackPropagation Through Time (Werbos, 1990).”
Cooperation and communication in multiagent deep reinforcement learning (2016, methodology): “Previously, LSTM networks have been demonstrated to solve POMDPs when trained using policy gradient methods (Wierstra et al., 2007).”
State-Dependent Exploration for Policy Gradient Methods (2008, methodology): “[13], we start with the probability of observing history hπ under policy π, which is given by the probability of starting with an initial observation x0, multiplied by the probability of taking action u0 under h0, multiplied by the probability of receiving the next observation x1, and so on.”
Chasing Ghosts: Instruction Following as Bayesian State Tracking (2019, unknown): “General purpose unstructured memory representations, such as LSTM memory [19], have been used extensively in both 2D and 3D environments [20–24].”
Learning deep neural network policies with continuous memory states (2015, methodology): “Finite state controllers have previously been applied to smaller RL tasks where value function approximation is practical [2], and policy gradient methods have been extended to recurrent neural networks (RNNs) [8].”
Neuroevolution strategies for episodic reinforcement learning (2009, methodology): “If after 10 evaluations no successful control strategy was found, a trial was regarded as a failure In such a case, the number of evaluations was set to 10 when computing the”