Policy gradient methods

By Jan Peters, J. Bagnell. 2010

142 citations recorded by the source corpus.

A policy gradient method is a reinforcement learning approach that directly optimizes a parametrized control policy by gradient descent. It belongs to the class of policy search techniques that maximize the expected return of a policy in a fixed policy class while traditional value function approximation approaches derive policies from a value function. Policy gradient approaches have various advantages: they allow the straightforward incorporation of domain knowledge in the policy parametrization and often significantly fewer parameters are needed for representing the optimal policy than the corresponding value function. They are guaranteed to converge to at least a locally optimal policy and can handle continuous states and action, and often even imperfect state information. Their major drawbags are the difficult use in off-policy settings, their slow convergence in discrete problems and that global optima are not attained.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

Citations in the corpus

Generalized Off-Policy Actor-Critic (2019, background): “In the on-policy continuing setting, we seek to optimize the average value objective (Silver, 2015)”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “1, except that C is now an FNN or RNN trained by evolutionary algorithms [200, 255, 105, 56, 68] applied to NNs [165, 321, 180, 259, 72, 90, 89, 110, 94], or by policy gradient methods [314, 315, 316, 274, 18, 1, 63, 128, 313, 210, 192, 191, 256, 85, 312, 190, 82, 93][245, Sec.”
One Big Net For Everything (2018, background): “6.6], or reinforcement learning [29, 90, 101, 76] through policy gradients [104, 91, 3, 1, 13, 30, 103, 48, 82, 22, 102, 42] [76, Sec.”
Online Multi-Task Learning for Policy Gradient Methods (2014, methodology): “…of state-action pairs forms a trajectory τ = [x 0:H , a 0:H ] over a possibly infinite horizon H. Policy gradient methods (Sutton et al., 1999; Peters & Schaal, 2008; Peters & Bagnell, 2010) have shown success in solving high-dimensional problems, such as robotic control (Peters & Schaal, 2007).”
Deterministic Policy Gradient Algorithms (2014, background): “It was previously believed that the deterministic policy gradient did not exist, or could only be obtained when using a model (Peters, 2010).”
DeepMimic (2018, methodology): “A popular class of algorithms for optimizing a parametric policy is policy gradient methods [Sutton et al. 2001], where the gradient of the expected return ▽θ J (θ ) is estimated with trajectories sampled by following the policy.”
Reward-based training of recurrent neural networks for cognitive and value-based tasks (2016, methodology): “65 In this work, we build on advances in recurrent policy gradient reinforcement learning, 66 specifically the application of the REINFORCE algorithm (Williams, 1992; Baird & Moore, 67 1999; Sutton et al., 2000; Baxter & Bartlett, 2001; Peters & Schaal, 2008) to recurrent neural 68 networks (RNNs) (”
Bias in Natural Actor-Critic Algorithms (2014, unknown): “…vt+1 = vt + α v t−τ 1 δte v t 15: Update Actor (Natural Policy Gradient): 16: if TYPE1, t − k ≥ 0, and (wt, w t−k ) ≤ or 17: TYPE2 and (count mod k = 0) then 18: θ ← θ + ητ 2 w t+1 ||w t+1 || 2 ; τ1 = t; τ2 = τ2 + 1 19: if st+1 terminal then break out of loop over t Bagnell, 2010; Peters, 2010 )).”