Technical Update: Least-Squares Temporal Difference Learning

By Justin A. Boyan. Machine-mediated learning, 2002

379 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/d6aa631d44a7f2cbb3ab07f3b993f4e5c5c4150b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

Citations in the corpus

Approximate Inference and Stochastic Optimal Control (2010, methodology): “1 The LSΨ algorithm Similar to numerous previous approaches [5, 20, 28, 29] we used a linear basis function model, to approximate Ψ, i.”
Incremental Off-policy Reinforcement Learning Algorithms (2017, methodology): “We call this algorithm the WIS-LSTD(λ) algorithm, because this is a generalization of LSTD(λ) algorithm (Boyan 2002) with WIS.”
Linear options (2010, background): “Least-Squares TD: Linear Least-Squares Temporal Difference learning for Markov decision processes [2, 1]—which we denote MDP-LSTD—achieves a lower sample complexity than TD learning by exploiting the following observation: given a fixed setting of the value function parameters θ, we can explicitly r”
Least-Squares Policy Iteration (2003, background): “It is also possible to extend LSTDQ to LSTDQ(λ) in a way that resembles closely LSTD(λ) (Boyan, 2002), but in that case it is necessary that the sample set consists of complete episodes generated using the policy under evaluation.”
Projected equation methods for approximate solution of large linear systems (2009, neutral): “Within the DP context, the approximation methods in (a) above have been proposed in [9] [8] (see also the analysis in [18]), and are known as least squares temporal differences (LSTD) methods.”
Tree-Based Batch Mode Reinforcement Learning (2005, neutral): “In Boyan (2002), the Least-Squares Temporal-Difference (LSTD) algorithm is proposed.”
Expected Policy Gradients (2017, methodology): “In every case, an LSTD-type critic [3] is used.”
Acceleration of Reinforcement Learning by Policy Evaluation Using Nonstationary Iterative Method (2014, methodology): “Recently, some methods of Temporal Difference (TD) learning with function approximation have been developed [2], [14], [15], [16], [17], [18], [19].”
Policy Gradients Beyond Expectations: Conditional Value-at-Risk (2014, neutral): “Obtaining an approximate value function for a given policy has been studied extensively in RL literature, and many efficient solutions for this task are known, such as LSTD (Boyan, 2002) and TD(λ) (Sutton & Barto, 1998).”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “LSTD (Boyan, 2002), TDC (originally named GTD(λ) in Maei, 2011, but here referred to as TDC(λ) for clarity), FPKF (Scherrer and Geist, 2011; Geist and Scherrer, 2013) and BRM (Scherrer and Geist, 2011; Geist and Scherrer, 2013) have been extended to use multistep-lookahead with eligibility traces.”
Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories (2012, methodology): “Batch mode RL techniques are probably rooted in the works of Bradtke and Barto (1996) and Boyan (2005) related to the use of least-squares techniques in the context of Temporal Difference learning methods (LSTD) for estimating the return of control policies.”
A Deeper Look at Planning as Learning from Replay (2015, methodology): “We call this method forgetful LSTD(λ) for obvious reasons: when the forgetting parameter β is set equal to 0, and Ainit = 0 and dinit = 0, the model 〈At,dt〉 reduces to the model learned by LSTD(λ) (Boyan, 2002).”
Gradient temporal-difference learning algorithms (2011, methodology): “Previous attempts to solve the off-policy problem • Importance sampling • With recognizers • Least-squares methods, LSTD, LSPI, iLSTD • Averagers • Residual gradient methods”
Learning the Variance of the Reward-To-Go (2016, methodology): “The first algorithm is a variant of the Least Squares Temporal Difference algorithm (LSTD; Boyan 2002), and aims to solve Eq.”
Zap Q-Learning (2017, background): “It is a fortunate coincidence that LSTD(λ) of [6] achieves this goal [8].”
Kalman Temporal Differences (2010, methodology): “This algorithm has also been extended to eligibility traces (for details, see Boyan, 1999).”
Improved empirical methods in reinforcement-learning evaluation (2015, background): “The two-stage modular learner is a least squares algorithm (Boyan, 2002) that minimizes Equation 3.”