Incremental Natural Actor-Critic Algorithms

By Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, Mark Lee. 2007

213 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The Fisher information matrix can be approximated by a block-diagonal matrix, where unit or quasi-diagonal natural gradients are used (Olivier, 2013).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “…[Witten, 1977, Barto et al., 1983] and many techniques since proposed to reduce variance of the estimates of this gradient [Konda and Tsitsiklis, 2000, Weaver and Tao, 2001, Greensmith et al., 2004, Peters et al., 2005, Bhatnagar et al., 2008, 2009, Grondman et al., 2012, Gu et al., 2016].”
Bayesian Reinforcement Learning (2010, background): “20, then the resulting mean squared error E π ( w ∗ ) is further minimized by setting b ( s ) = V π ( s ) (Bhatnagar et al, 2007, 2009).”
Expected Policy Gradients (2017, methodology): “Instead of learning Q̂, we can set b(s) = −V (s) so that Q(a, s) + b(s) = A(s, a) and then use the TD error δ(r, s′, s) = r + γV (s′) − V (s) as an estimate of A(s, a) (Bhatnagar et al. 2008):”
Kalman Temporal Differences (2010, methodology): “As said before, KTD could be an interesting alternative to TD as the actor part of the incremental natural actor-critic algorithms of Bhatnagar et al. (2008). Some preliminary works on using KTD in an actor-critic architecture are provided by Geist and Pietquin (2010c). Talking about natural gradien”
AuTO: scaling deep reinforcement learning for datacenter-scale automatic traffic optimization (2018, methodology): “Same as DPG, DDPG is also an actor-critic [12] algorithm, and it maintains four DNNs.”
Stochastic variational learning in recurrent spiking networks (2014, background): “Stochastic gradients of the form Equation (25) have been extensively studied in the reinforcement learning literature and several approaches with different levels of complexity have been proposed for reducing their variance (Munos, 2005; Bhatnagar et al., 2007).”
Recurrent policy gradients (2010, background): “…an exception, as these allow for learning policies even with noisy state information [3], work in combination with function approximation [4, 5], are compatible with policies that have internal memory [6], can naturally deal with continuous actions [7–9] and are guaranteed to converge at…”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, methodology): “Convergence of actor-critic and natural actor-critic algorithms with linear function approximation have been studied in Bhatnagar et al. (2009, 2008); Castro and Meir (2010); Bhatnagar (2010).”
Real-time reinforcement learning by sequential Actor-Critics and experience replay (2009, methodology): “Several important RL algorithms, such as QLearning (Watkins & Dayan, 1992) and Actor-Critic methods (Barto et al., 1983; Kimura & Kobayashi, 1998; Konda & Tsitsiklis, 2003; Bhatnagar et al., 2008a), Preprint process the data sequentially.”
Convex Optimization: Algorithms and Complexity (2015, unknown): “A policy is called stationary if it does not change over time.”
Natural actor-critic algorithms (2009, result): “Our results are an extension of our prior work [24], and of prior work on the convergence of two-timescale stochastic approximation recursions [1, 21, 45, 46].”
Expected Policy Gradients for Reinforcement Learning (2018, methodology): “…b ( s ) = − V ( s ) so that Q ( a, s ) + b ( s ) = A ( s, a ) and then use the TD error δ ( r, s (cid:48) , s ) = r + γV ( s (cid:48) ) − V ( s ) as an estimate of A ( s, a ) (Bhatnagar et al., 2008): where ˆ V ( s ) is an approximate value function learned using any policy evaluation algorithm.”
Algorithms for CVaR Optimization in MDPs (2014, background): “We can show that is the temporal-difference (TD) error in ¯ M , and (cid:98) V is an unbiased estimator of V θ (see e.g., [6, 7]).”
Deterministic Policy Gradient Algorithms (2014, methodology): “…ii) is usually relaxed in favour of policy evaluation algorithms that estimate the value function more ef-ﬁciently by temporal-difference learning (Bhatnagar et al., 2007; Degris et al., 2012b; Peters et al., 2005); indeed if both i) and ii) are satisﬁed then the overall algorithm is equivalent…”