Analytical Mean Squared Error Curves for Temporal Difference Learning

By Satinder Singh, Peter S. Dayan. 1996

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Q(λ) with Off-Policy Corrections (2016, background): “It is well-known that λ trades off the bias from bootstrapping with an approximate Q-function, with the variance from using a sampled multi-step return [4], with intermediate values of λ usually performing best in practice [15, 13].”
Temporal-difference search in computer Go (2012, unknown): “In reinforcement learning, bootstrapping often provides a substantial reduction in variance and an improvement in performance (Singh and Dayan 1998; Sutton and Barto 1998).”
Performance bounds for λ policy iteration and application to the game of Tetris (2013, unknown): “…algorithms value iteration (which corresponds to the case λ = 0) and policy iteration (case λ = 1), and has some deep connections with the temporal- Scherrer difference algorithms that are well known to the reinforcement-learning community (Sutton and Barto, 1998; Bertsekas and Tsitsiklis, 1996).”
Natural actor-critic algorithms (2009, methodology): “Such “bootstrapping methods” [62] can be viewed as a way of accelerating learning by trading bias for variance [57].”
Reinforcement learning for factored Markov decision processes (2002, result): “These results are consistent with previous results on learning with eligibility traces [Singh and Sutton 1996; Singh and Dayan 1998; He and Shayman 1999], although the specific best value for λ varies for different problems in the literature.”
A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning (2016, methodology): “Singh and Dayan [18] and Kearns and Singh [5] contributed extensive simulation studies of the interaction between λ and other agent parameters on a chain MDP, but again relied on access to the model and offline computation.”
Q( λ ) with Oﬀ-Policy Corrections (background): “It is well-known that λ trades oﬀ the bias from bootstrapping with an approximate Q-function, with the variance from using a sampled multi-step return [4], with intermediate values of λ usually performing best in practice [15, 13].”
The optimal unbiased value estimator and its relation to LSTD, TD and MC (2009, background): “The issue of bias and variance in reinforcement learning is also addressed in other works Singh and Dayan (1998) , Mannor et al. (2007).”