OnActor-Critic Algorithms

By Vijay R. Konda, John N. Tsitsiklis. SIAM Journal of Control and Optimization, 2003

360 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c1bfa4cd9bb82610d67c13aca57cb46437fae34f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Nando de Freitas
4. Hado van Hasselt
5. Shimon Whiteson
6. Shie Mannor
7. Amir-massoud Farahmand
8. Csaba Szepesvari

In Control and Systems Engineering

1. Amir-massoud Farahmand
2. Hado van Hasselt
3. Pawel Wawrzynski
4. Csaba Szepesvari
5. Shie Mannor
6. A. Srinivas
7. Kenji Doya
8. Kristian Kersting

In Statistics and Probability

1. Amir-massoud Farahmand
2. Pawel Wawrzynski
3. Shie Mannor
4. Hado van Hasselt
5. Csaba Szepesvari
6. Balaraman Ravindran
7. Kurt Driessens
8. Kenji Doya

Citations in the corpus

• Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “Algorithms that only store a policy, and not a value function, are often calledirect policy-search(Ng et al, 1999) oractor-only algorithms (Konda and Tsitsiklis, 2003).”
• Autonomous reinforcement learning with experience replay. (2013, background): “A broad class of RL algorithms that encompass Actor–Critics (Bhatnagar, Sutton, Ghavamzadeh, & Lee, 2009; Konda & Tsitsiklis, 2003; Peters, Vijayakumar, & Schaal, 2005) and policy gradient methods (Sutton, McAllester, Singh, & Mansour, 2000; Williams, 1992) optimizes a control policy on the basis…”
• Convergence rate of linear two-time-scale stochastic approximation (2004, background): “Over the past few years, several such algorithms have been proposed for various applications [Konda and Borkar (1999), Bhatnagar, Fu, Marcus and Fard (2001), Baras and Borkar (2000), Bhatnagar, Fu and Marcus (2001) and Konda and Tsitsiklis (2003)].”
• An Actor-Critic based controller for glucose regulation in type 1 diabetes (2013, methodology): “The Critic’s basis functions φ are chosen to be equal to the vector which is the most straightforward approach in AC algorithms [34].”
• Regularization in reinforcement learning (2011, methodology): “Moreover, there are hybrid methods that explicitly represent both value and policy functions [Konda and Tsitsiklis, 2001; Peters et al., 2003; Ghavamzadeh and Engel, 2007a].”
• Constrained reinforcement learning from intrinsic and extrinsic rewards (2007, methodology): “We also use a simplified method based on the actor critic method (Konda & Tsitsiklis, 2003) that exploits a value function.”
• The Cross Entropy Method for Fast Policy Search (2003, methodology): “We assume that the policy is parameterized by a small number of parameters (as in Konda & Tsitsiklis, 2003).”
• A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot (2009, methodology): “However, since the dynamic model is non-differentiable, one cannot use gradient-based stochastic approximations [Konda and Tsitsiklis (2003); Singh et al (2005)] or policy gradient methods [Baxter and Bartlett (2001); Peters and Schaal (2008a,b)] to update the parameters of the policy.”
• Dynamic Action Repetition for Deep Reinforcement Learning (2017, methodology): “In contrast, Actor-Critic methods (Konda and Tsitsiklis 2003) explicitly model the policy that is executed by the agent along with a value function component updated using Temporal Difference (TD) learning methods.”