The convergence of TD(λ) for general λ

By Peter S. Dayan. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Q-learning [140] is an off-policy TD algorithm which is proved to converge [141, 20].”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Watkins (1989) extended TD learning to control in the form of Q-learning and proved its convergence in the tabular case (without function approximation, Watkins & Dayan 1992), while Rummery (1995) extended TD learning to control in an on-policy form as the Sarsa(λ) algorithm.”
Reinforcement Learning with Replacing Eligibility Traces (2005, methodology): “Although this algorithm is indeed identical to TD( ), the theoretical results for TD( ) onstationary prediction problems (e.g., Sutton, 1988; Dayan, 1992) do not apply here becausethe policy is continually changing, creating a nonstationary prediction problem.7.”
A Tour of Reinforcement Learning: The View from Continuous Control (2018, methodology): “In particular, Temporal Diﬀerence algorithms are derived from the value-function-centric perspective [75, 26, 21, 13, 89].”
Practical Issues in Temporal Difference Learning (1991, background): “This proof has recently been extended to arbitrary values of), in Dayan (1992).”
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms (2000, background): “Keywords: reinforcement-learning, on-policy, convergence, Markov decision processes”
Analytical Mean Squared Error Curves for Temporal Difference Learning (1996, methodology): “The following summary hypotheses for TD algorithms can be formulated from the data shown in Figures 7 and 8: H1 For a fixed Markov reward process and a constant λ, increasing α has two general effects on the learning curve: there is a largest value of α below which the bias converges to zero and abo”
True Online Temporal-Difference Learning (2015, background): “The error for accumulate TD( λ ) converges to the least mean squares (LMS) error for λ = 1, as predicted by the theory (Dayan, 1992).”
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules (2016, background): “(11) If the estimated state values are updated using the information contained in δTD, the estimated state values will eventually converge to the true solution (Dayan, 1992).”
Online Transfer Learning in Reinforcement Learning Domains (2015, methodology): “In this case, the convergence of Q-learning, Sarsa, and other related algorithms (such as TD( λ )) have been shown by previous work Peter (1992); Watkins and Dayan (1992); Singh et al. (2000).”
Self-improving reactive agents based on reinforcement learning, planning and teaching (1992, background): “2. See (Lin, 1991c); Watkins, 1989; Dayan, 1992 ) for using TD(X), k > 0. 3. For simplicity, I will assume the world is deterministic throughout the discussion of the learning algorithms, although it is straightforward to extend the discussion to handle nondeterministic worlds (Barto et al., 1990; ”
Reinforcement learning for closed-loop propofol anesthesia: a study in human volunteers (2014, methodology): “Q-learning is mathematically robust (Tsitsiklas and Van Roy, 1996; Dayan, 1992), and this robustness has contributed to the method's popularity in applied reinforcement learning.”
TD(λ) Converges with Probability 1 (1994, methodology): “That the simplest form of TD converges with probability one in a very special case was shown by Dayan (1992) .”
Reinforcement Learning: A Tutorial Survey and Recent Advances (2009, background): “Several other papers (Gordon, 1995; Hauskrecht, 2000; Dayan, 1992; Tsitsiklis and van Roy, 1996) discuss the issue of function approximation.”