A Natural Policy Gradient

By Sham M. Kakade. 2001

1,429 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, methodology): “The algorithm has also been successfully applied in the reinforcement learning community (Kakade, 2001; Peters and Schaal, 2008) and for stochastic search (Sun et al. , 2009).”
Trust Region Policy Optimization (2015, methodology): “The ﬁrst proof extends Kakade and Langford’s result using the fact that the random variables from two distributions with total variation divergence less than α can be coupled, so that they are equal with probability 1 − α .”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Truncated Natural Policy Gradient (TNPG) (Kakade, 2002; Peters et al., 2003; Bagnell & Schneider, 2003; Schulman et al., 2015a): Natural Policy Gradient improves upon REINFORCE by computing an ascent direction that approximately ensures a small change in the policy distribution.”
High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “The idea of compatible features motivates an elegant method for computing the natural policy gradient (Kakade, 2001a; Peters & Schaal, 2008).”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “Policy gradients were introduced in (Sutton et al., 1999) and have been used extensively (Kakade, 2001; Peters and Schaal, 2006; Deisenroth et al., 2011).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “FActorized Natural Gradient (FANG) (Grosse & Salakhudinov, 2015) estimates the natural gradient via an approximation of the Fisher information matrix by a Gaussian graphical model.”
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, neutral): “[22, 32, 44, 48]).”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “Well-known examples include Trust Region Policy Optimization (TRPO) (Schulman et al., 2015), and use conservative, KL-regularized policy updates to restrict the RL algorithm to stay close to its own prior policy (Haarnoja et al., 2018; Kakade, 2002; Peters et al., 2010; Rawlik et al., 2012).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “Previous work (Kakade, 2002; Bagnell & Schneider, 2003; Peters & Schaal, 2008; Schulman et al., 2015) has applied natural gradient to policy optimization, locally improving expected reward subject to variants of d θ T F ( θ )d θ ≤ (cid:15) .”
Equivalence Between Policy Gradients and Soft Q-Learning (2017, background): “As pointed out by Kakade [2002], the natural gradient step can be computed as the solution to a least squares problem.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “Although policy updates can be improved by the use of appropriate geometry [14, 27, 32], the need for variance reduction remains paramount.”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “…are desirable for exploration, this exploration is typically attained heuristically, for example by injecting noise (Silver et al., 2014; Lillicrap et al., 2015; Mnih et al., 2015) or initializing a stochastic policy with high entropy (Kakade, 2002; Schulman et al., 2015a; Mnih et al., 2016).”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, background): “Therefore, Q-Prop can be directly combined with a number of prior techniques from both on-policy methods such as natural policy gradient (Kakade, 2001), trust-region policy optimization (TRPO) (Schulman et al., 2015) and generalized advantage estimation (GAE) (Schulman et al., 2016), and off-policy…”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “In policy gradient techniques the policy is represented explicitly and we improve the policy by updating the parameters in the direction of the gradient of the performance (Sutton et al., 1999; Silver et al., 2014; Kakade, 2001).”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “For planning (ii): one can use upper confidence bound tree search (UCT) [27] or policy gradient methods [24, 42].”
Neural Approaches to Conversational AI (2018, neutral): “rk on how to compute the gradient r Jthat is more effective than the steepest descent in Eqn. 2.7. Interested readers can refer to a few related works and the references therein for further details (Kakade, 2001; Peters et al., 2005; Schulman et al., 2015a,b; Mnih et al., 2016; Gu et al., 2017; Dai ”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “For a full derivation of this gradient, we refer the reader to prior work (Sutton et al., 2000; Kakade, 2002; Schulman et al., 2015a).”
Efficient Architecture Search by Network Transformation (2017, methodology): “Net2Deeper Actor Network Figure 1: Overview of our proposed framework. Similar to [35], we use a neural network to represent the policy of the agent and train the network with policy gradient methods [33,13,21]. Our work is also related to meta-learning or learning to learn [31], the idea of using a”