Natural actor-critic algorithms

By Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, Mark Lee. 2009

649 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, methodology): “Direct policy search can be performed in three main ways: (1) by using black-box local search methods, such as hill climbing or the NelderMead simplex technique [34]; (2) by using gradient descent methods [86, 6, 72, 9]; or (3) by using EM or crossentropy approaches [31, 46].”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “deling of the behavior policy, AWAC avoids overly conservative updates. 5Related Work Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied extensively [19, 5, 25, 11, 7, 4, 32, 48, 44, 3]. While standard off-policy methods are able to beneﬁt from incl”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, methodology): “E − − Control variates have been used before in reinforcement learning to reduce the variance of policy gradient estimates (Bhatnagar et al., 2009), where the control variate was referred to as a baseline .”
Deep Reinforcement Learning and the Deadly Triad (2018, methodology): “Bootstrapping is also commonly used within policy-gradient and actor-critic methods to learn value functions (Witten, 1977; Barto et al., 1983; Peters and Schaal, 2008; Bhatnagar et al., 2009; van Hasselt, 2012; Degris et al., 2012; Mnih et al., 2016).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, neutral): “L G ] 2 2 N ov 2 Bhatnagar et al., 2008, 2009, Grondman et al., 2012, Gu et al., 2016].”
Linear Off-Policy Actor-Critic (2012, methodology): “Given that we have satisﬁed requirements 1-5 and given our step-size conditions, using standard arguments (c.f. Lemma 6 in Bhatnagar et al., 2009), we can deduce that u t converges almost surely to the set of asymptotically stable ﬁxed points, ˜ Z , of ˙ u = ˆΓ g ( u ).”
Policy Networks with Two-Stage Training for Dialogue Systems (2016, background): “Such algorithms are known in general as actor-critic algorithms, where the Q approximator is the critic and π θ is the actor (Sutton, 1984; Barto et al., 1990; Bhatnagar et al., 2009).”
Dynamic policy programming (2010, background): “The gradient update ensures that PGAC asymptotically converges to a local maximum, given that an unbiased estimate of the gradient is provided by the critic (Maei et al., 2010; Bhatnagar et al., 2009; Konda and Tsitsiklis, 2003; Kakade, 2001).”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, methodology): “ote by and Y, the projection into sets and Y, respectively. This is standard in stochastic approximation algorithms and has been used in off-policy TD() [Yu, 2012] and actor-critic algorithms (e.g., Bhatnagar et al. 2009). 2) after niterations (nis the number of training samples in D), the algorithm”
Convex Optimization: Algorithms and Complexity (2015, neutral): “A policy is called stationary if it does not change over time.”
Safe Reinforcement Learning (2015, methodology): “Although such policy search methods have been investigated (Bhatnagar et al., 2009, Thomas et al., 2013), like the controltheoretic approaches, they require an approximate model of the system in order to be applicable (to determine the safe region of policy space).”
Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation (2018, methodology): “Also see, Bhatnagar et al., 2009; Konda & Tsitsiklis; 2003).”
Reward Design via Online Gradient Ascent (2010, methodology): “We develop PGRD, presented in Figure 1, explicitly as a generalization of OLPOMDP, a policy gradient algorithm developed by Bartlett and Baxter [2], because of its foundational simplicity relative to other policy-gradient algorithms such as those based on actor-critic methods (e.g., [4]).”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, methodology): “There are many natural actor-critics, including Natural policy gradient utilizing the Temporal Differences (NTD) algorithm [99], Natural Actor-Critic using LSTD-Q(λ) (NAC-LSTD) [100], Episodic Natural Actor-Critic (eNAC) [100], Natural Actor-Critic using Sarsa(λ) (NAC-Sarsa) [101], Incremental Natur”
A Survey of Optimization Methods From a Machine Learning Perspective (2019, methodology): “alue function of the current policy. That is to evaluate the quality of the actor. In the actor-critic method, the critic solves a problem of prediction, while the actor pays attention to the control [176]. There is more information of actor-critic method in [75], [176] The summary of the value-base”
Learning Action Representations for Reinforcement Learning (2019, methodology): “Now using sequence of approximations using Gronwall’s inequality, it can be shown that (21) converges to Z ∗ asymptotically [Borkar, 2009].”