Actor-Critic Algorithms

By Vijay R. Konda, John N. Tsitsiklis. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “Previous actor critic methods, e.g. in Konda & Tsitsiklis (2003), use a Q-function to obtain potentially low-variance policy gradient estimates.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, methodology): “Promising candidates include further reducing the variance using a learned value function (Konda & Tsitsiklis, 2000) and preventing detrimentally large policy updates by maintaining a trust region in the policy space – akin to techniques such as TRPO (Schulman et al., 2015) and PPO (Schulman et…”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Function approximation to learn a “critic” instead of using rewards directly has been embraced in the RL literature under the name of “actor-critic” methods (see Konda and Tsitsiklis (2000), Sutton et al. (2000), Kakade (2001), Schulman et al. (2015), Mnih et al. (2016), inter alia).”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “This creates a two-timescale algorithm, as often required for convergence in the linear setting (Konda & Tsitsiklis, 2003).”
Neural Approaches to Conversational AI (2018, neutral): “g the entire trajectory. Its variance may be reduced by the use of an estimated value function of the current policy, often referred to as the critic in actor-critic algorithms (Sutton et al., 1999a; Konda and Tsitsiklis, 1999): r J() = HX 1 t=1 t 1 r logˇ(a tjs t;)Q^(s t;a t;h) ; (2.8) where Q^(s”
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium (2017, methodology): “(C1) The proofs only use the boundedness of the moments of Hn [31, 33], therefore Hn may depend on wn.”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “For this reason, continuous domains are often tackled using actor-critic methods (Konda & Tsitsiklis, 1999; Hafner & Riedmiller, 2011; Silver et al., 2014; Lillicrap et al., 2016), where a separate parameterized “actor” policy π is learned in addition to the Q-function or value function “critic,”…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “There are a number of different variants of actor-critic methods, including on-policy variants that directly estimate V (s) (Konda and Tsitsiklis, 2000), and off-policy variants that estimate Q(s,a) via a parameterized state-action value function Qφ(s,a) (Haarnoja et al.”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “deling of the behavior policy, AWAC avoids overly conservative updates. 5Related Work Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied extensively [19, 5, 25, 11, 7, 4, 32, 48, 44, 3]. While standard off-policy methods are able to beneﬁt from incl”
Counterfactual Multi-Agent Policy Gradients (2017, background): “COMA takes an actor-critic (Konda and Tsitsiklis 2000) approach, in which the actor, i.”
Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “In this case, actor-critic methods are commonly used, where action selection is performed through a separate policy network π φ , known as the actor, and updated with respect to a value estimate, known as the critic (Sutton & Barto, 1998; Konda & Tsitsiklis, 2003).”
Self-Imitation Learning (2018, background): “Experience replay for actor-critic In fact, actor-critic framework (Sutton et al., 1999; Konda & Tsitsiklis, 2000) can also utilize experience replay.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “, 2015) and actor-critic algorithms (Haarnoja et al., 2017; Konda and Tsitsiklis, 2002), is a pathological interaction between the data distribution induced by the latest policy, and the errors induced in the learned approximate value function as a consequence (1)UC Berkeley.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “The traditional max-return objective is in the form of an on-policy expectation, and thus, policy gradient methods (Sutton et al., 2000; Konda and Tsitsiklis, 2000) require samples from the on-policy distribution to estimate the gradient of this objective.”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, methodology): “For example, they have straightforward update rules and convergence guarantee, as they minimizes J directly by updating parameters in descent directions in expectation [2, 7–9].”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “…actor-critic algorithms [Witten, 1977, Barto et al., 1983] and many techniques since proposed to reduce variance of the estimates of this gradient [Konda and Tsitsiklis, 2000, Weaver and Tao, 2001, Greensmith et al., 2004, Peters et al., 2005, Bhatnagar et al., 2008, 2009, Grondman et al., 2012,…”