Temporal-difference methods and Markov models

By Etienne Barnard. 1993

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, background): “successor state s0˘P ˇ(js), and a step-size parameter >0, this update is + r ˇ(s) + ˚(s0)> ˚(s)> ˚(s): (2) While (2) does not correspond to a proper gradient descent procedure (see e.g. Barnard 1993), it can be shown to converge, as we shall now see. The expected behavior of the semi-gradi”
Reinforcement Learning with Replacing Eligibility Traces (2005, methodology): “Batch TD(1) is a gradient-descentprocedure known to converge to the estimate with minimum mean squared erroron the training set (Sutton, 1988; Dayan, 1992; Barnard, 1993).”
Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation (2009, unknown): “However, the simplest and most popular methods, including TD(λ), Q-learning, and Sarsa, are not true gradient-descent methods (Barnard 1993) and, as a result, the conditions under which they converge are narrower and less robust than can usually be guaranteed for gradient-descent methods.”
An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time (2013, background): “…was thought that the VGL weight updates (equation (4), and DHP and GDHP) were based on gradient descent on an error function (cid:17) But as [21] showed, the TD( λ ) weight update is not true gradient descent on its intended error function, and it is not gradient descent on any error function [25].”
Gradient temporal-difference learning algorithms (2011, background): “A good way to see why linear/nonlinear TD(0) is not a true gradient-descent method is to show that its update can not be derived by taking gradient of any function (Barnard, 1993).”
Reinforcement Learning: An Introduction (1998, methodology): “Bootstrapping methods are not in fact instances of true gradient descent (Barnard, 1993).”
Analytical Mean Squared Error Curves for Temporal Difference Learning (1996, background): “Tsitsiklis (1994), Barnard (1993) ; and for convergence of replace TD, first-visit MC, and everyvisit MC, see Singh & Sutton (1996).”
Value-gradient learning (2012, background): “Furthermore, it was proven by Barnard (1993) that TD(0) is not gradient descent on any error function, for if Eq.”
Reinforcement learning with replacing eligibility traces (2004, methodology): “Batch TD(1) is a gradient-descent procedure known to converge to the estimate with minimum mean squared error on the training set (Sutton, 1988; Dayan, 1992; Barnard, 1993).”
Fast gradient-descent methods for temporal-difference learning with linear function approximation (2009, unknown): “However, the simplest and most popular methods, including TD(λ), Q-learning, and Sarsa, are not true gradient-descent methods (Barnard 1993) and, as a result, the conditions under which they converge are narrower and less robust than can usually be guaranteed for gradient-descent methods.”
True online TD(λ) (2014, background): “Although TD(λ) is not strictly speaking a gradient-descent method (Barnard, 1993) it is still useful to view it in these terms (as in Sutton & Barto, 1998).”