TD(λ) Converges with Probability 1

By Peter S. Dayan, Terrence J. Sejnowski. 1994

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, background): “…the usual Robbins-Munro conditions: for each state x , The proof is a straightforward application of the convergence result for stochastic approximation algorithms to the ﬁxed point of a contraction operator, see e.g. Dayan & Sejnowski (1994); Bertsekas & Tsitsiklis (1996); Kushner & Yin (2003).”
Reinforcement Learning with Replacing Eligibility Traces (2005, methodology): “We then analyze the relative e ciency of the two Monte Carlo methods.”
Analytical Mean Squared Error Curves for Temporal Difference Learning (1996, background): “4. For convergence of accumulate TD, see Dayan & Sejnowski (1994) , Jaakkola et al. (1994), Tsitsiklis (1994),”
Reinforcement learning with replacing eligibility traces (2004, methodology): “Systematic empirical studies of eligibility traces in conjunction with TD methods were made by Sutton (1984), and theoretical results have been obtained by several authors (e.g., Dayan, 1992; Jaakkola, Jordan & Singh, 1994; Tsitsiklis, 1994; Dayan & Sejnowski, 1994; Sutton & Singh, 1994).”
Integrating a Partial Model into Model Free Reinforcement Learning (2012, methodology): “A similar analysis of TD(0) without partial model was given by Dayan and Sejnowski (1994) for a decreasing step size and without explicit convergence rate results, and by Konda (2002) using a similar technique to bound the convergence rate.”
Temporal-difference search in computer Go (2012, unknown): “…heuristic with a frequency heuristic: states which are visited most frequently and most recently are given the greatest eligibility (Sutton 1984), If all states are visited infinitely many times, and with appropriate step-sizes, TD(λ) converges to V π for all values of λ (Dayan and Sejnowski 1994).”
The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. (2002, background): “Several studies have shown that the algorithm converges rapidly under many conditions (e.g., Dayan & Sejnowski, 1994; Jaakkola, Jordan, & Singh, 1994; Sutton, 1988).”
Reinforcement Learning in Continuous State and Action Spaces (2012, background): “Applied to reinforcement learning, this has led to a number of convergence guarantees, under various additional assumptions (Sutton, 1984, 1988; Dayan, 1992; Peng, 1993; Dayan and Sejnowski, 1994; Bertsekas and Tsitsiklis, 1996; Tsitsiklis and Van Roy, 1997).”
Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms (2000, background): “…and the resulting convergence more formally, we note first that due to the dependence on the learning policy,SARSA(0) does not directly fall under the previously published convergence theorems (Dayan & Sejnowski, 1994; Jaakkola et al., 1994; Tsitsiklis, 1994; Szepesv´ari & Littman, 1996).”