Natural Actor-Critic

By Jan Peters, Sethu Vijayakumar, Stefan Schaal. 2003

1,274 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, methodology): “The algorithm has also been successfully applied in the reinforcement learning community (Kakade, 2001; Peters and Schaal, 2008) and for stochastic search (Sun et al. , 2009).”
Trust Region Policy Optimization (2015, methodology): “In contrast, the single path algorithm requires no state resets and can be directly implemented on a physical system (Peters & Schaal, 2008b).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “The Fisher information matrix can be approximated by a block-diagonal matrix, where unit or quasi-diagonal natural gradients are used (Olivier, 2013).”
Neural Approaches to Conversational AI (2018, neutral): “ompute the gradient r Jthat is more effective than the steepest descent in Eqn. 2.7. Interested readers can refer to a few related works and the references therein for further details (Kakade, 2001; Peters et al., 2005; Schulman et al., 2015a,b; Mnih et al., 2016; Gu et al., 2017; Dai et al., 2018a;”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “deling of the behavior policy, AWAC avoids overly conservative updates. 5Related Work Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied extensively [19, 5, 25, 11, 7, 4, 32, 48, 44, 3]. While standard off-policy methods are able to beneﬁt from incl”
Reinforcement learning in robotics: A survey (2013, methodology): “…of additional structure within the reinforcement learning domain, including, for instance, the (approximate) Markov structure of problems, developing approximate models, value-function estimates when available (Peters and Schaal, 2008c), or even simply the causal ordering of actions and rewards.”
Active learning of inverse models with intrinsically motivated goal exploration in robots (2013, neutral): “…: when a goal is set, an optimization procedure can improve the parameters of the action policy to reach the goal, such as policy gradient methods [98,84] or stochastic optimization [99]; A optional feature, which is a variant of the second assumption above, is: • Active optimization :…”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, methodology): “The goal of this paper is to improve the learning performance of policy gradient methods [1, 7, 2– 4, 13, 5, 6].”
Deep Reinforcement Learning and the Deadly Triad (2018, methodology): “Bootstrapping is also commonly used within policy-gradient and actor-critic methods to learn value functions (Witten, 1977; Barto et al., 1983; Peters and Schaal, 2008; Bhatnagar et al., 2009; van Hasselt, 2012; Degris et al., 2012; Mnih et al., 2016).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “…[Witten, 1977, Barto et al., 1983] and many techniques since proposed to reduce variance of the estimates of this gradient [Konda and Tsitsiklis, 2000, Weaver and Tao, 2001, Greensmith et al., 2004, Peters et al., 2005, Bhatnagar et al., 2008, 2009, Grondman et al., 2012, Gu et al., 2016].”
Boosting the Actor with Dual Critic (2017, methodology): “However, the update directions are computed with the help of the critic, which can be more eﬃciently learned as in value-function-based methods (Sutton et al., 2000; Konda and Tsitsiklis, 2003; Peters et al., 2005; Bhatnagar et al., 2009; Schulman et al., 2015b).”
Linear Off-Policy Actor-Critic (2012, background): “For example, the nat-ural actor-critic, an on-policy policy-gradient algo-rithm, has been successful for learning in continuous action spaces in several robotics applications (Peters & Schaal, 2008).”
Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations (2017, methodology): “A number of pre-conditioned policy gradient methods have been developed in literature [19], [4], [35], [34], [43], [40], [44] and in principle any of them could be used.”
Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation (2017, background): “Demonstrations could instead be collected from running trajectory optimization [12], [13], [14], [15] or reinforcement learning [16], [17], [18], [19], [20], [21], [22], but these meth-ods require well-shaped, carefully designed reward functions, access to dynamics model, and substantial robot…”
Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost (2018, methodology): “There are numerous variants of the natural policy gradient method [28], [29].”
The Option-Critic Architecture (2016, background): “Levy and Shimkin [2011] adapted the Natural Actor-Critic [Peters et al., 2005] framework to learning options by taking a literal approach to the augmented state space construction.”
New Insights and Perspectives on the Natural Gradient Method (2014, methodology): “It has been successfully applied to a variety of problems such as blind source separation (Amari and Cichocki, 1998), reinforcement learning (Peters and Schaal, 2008), and neural network training (e.g. Park et al., 2000; Martens and Grosse, 2015; Desjardins et al., 2015).”