An Analysis of Actor/Critic Algorithms Using Eligibility Traces: Reinforcement Learning with Imperfect Value Function

By Hajime Kimura, Shigenobu Kobayashi. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “While the formula has been proposed in prior work [8, 19], our analysis is novel and enables GAE to be applied with a more general set of algorithms, including the batch trust-region algorithm we use for our experiments.”
Counterfactual Multi-Agent Policy Gradients (2017, background): “Perhaps the simplest form of policy gradient is REINFORCE (Williams, 1992), in which the gradient is: (1) In actor-critic approaches (Kimura et al., 2000; Schulman et al., 2015; Wang et al., 2016; Hafner & Riedmiller, 2011), the actor , i.e., the policy, is trained by following a gradient that…”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “For example, function approximators (critics) have been adopted to (partially) replace the Monte Carlo estimates of accumulated costs to reduce variance but at the expense of introducing bias in the search direction [2, 11–15].”
Infinite-Horizon Policy-Gradient Estimation (2001, methodology): “Marbach and Tsitsiklis (1998) provide the only convergence proof that we know of, albeit for a slightly different update of the form θt+1 = θt + γt [r(Xt, s)− η̂(θt)] zt, where η̂(θt) is a moving estimate of the expected performance, and is also updated on-line (this update was first suggested in th”
Autonomous reinforcement learning with experience replay. (2013, background): “This section presents actor–critic algorithms (Barto, Sutton, & Anderson, 1983; Bhatnagar et al., 2009; Kimura & Kobayashi, 1998; Konda & Tsitsiklis, 2003) and the way of its speeding-up by experience replay (Adam et al., 2012; Cichosz, 1999; Wawrzyński, 2009).”
Variance Reduction Techniques for Gradient Estimates in Reinforcement Learning (2001, background): “In Section 5, we describe the results of preliminary experiments, which show that these algorithms give performance improvements.”
Learning CPG-based biped locomotion with a policy gradient method (2005, methodology): “However, Kimura et al. [8] suggested that the approximated value function can be useful to reduce the variance of the gradient estimation in (13) even if the consistency condition in (10) is not satisfied.”
Learning CPG Sensory Feedback with Policy Gradient for Biped Locomotion for a Full-Body Humanoid (2005, methodology): “Although it is computationally infeasible to calculate the optimal value function for the POMDP by using dynamic programming in high-dimensional state space (Kaelbling, Littman, & Cas-sandra 1998), we can still attempt to ﬁnd a locally optimal policy by using policy gradient methods (Kimura & AAAI-0”
Learning CPG-based Biped Locomotion with a Policy Gradient Method: Application to a Humanoid Robot (2008, methodology): “Because we consider a learning framework in POMDPs, i.e., we only observe partial states x from all states x all , the TD-error usually does not converge to zero.”
Real-time reinforcement learning by sequential Actor-Critics and experience replay (2009, methodology): “Let us analyze the first example of methods that will be of interest for us: the algorithm presented by Kimura & Kobayashi (1998), quoted in the table beside, and called here the Basic Actor-Critic.”
A Cat-Like Robot Real-Time Learning to Run (2009, methodology): “ActorCritics [2,5,6,3] constitute probably the most efficient and the most theoretically developed class of reinforcement learning algorithms.”
Learning to Walk in 20 Minutes (2005, methodology): “The form of the algorithm that we present here was originally proposed by [14].”