Least-Squares Temporal Difference Learning

By Justin A. Boyan. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, methodology): “When making linear predictions with least-squares temporal-difference learning [LSTD, Bradtke and Barto, 1996, Boyan, 1999], the model-free algorithm on the original data does not require (or indeed benefit from) planning: the solution will already be a best fit (in a least squares sense) even with…”
Fitted Q-Learning for Relational Domains (2020, methodology): “With RRTs for computing gradients at hand, we now present the algorithm for learning the approximate Q-functions of relational MDPs using RFGB. Speciﬁcally, Algs 1 and 2 present the outline of our proposed approach, called Relational Gradient-Boosted Q-Learning (GBQL).”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “We implemented Boyan’s Chain (Boyan, 1999) by ourselves.”
The Uncertainty Bellman Equation and Exploration (2017, background): “For any fixed dataset we can find the least squares solution for each action a (Boyan, 1999), minimizewa ∑N i=1(φ(si) wa − yi)(2)2,”
On Convergence of Emphatic Temporal-Difference Learning (2015, background): “, (Boyan, 1999; Yu, 2012) for on-policy and off-policy LSTD(λ)).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Watkins (1989) extended TD learning to control in the form of Q-learning and proved its convergence in the tabular case (without function approximation, Watkins & Dayan 1992), while Rummery (1995) extended TD learning to control in an on-policy form as the Sarsa(λ) algorithm.”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, methodology): “We are not the ﬁrst to study the connection between model-free and model-based methods, with Boyan (1999) and Parr et al. (2008) being two notable examples.”
Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping (2008, methodology): “Others have proposed linear least-squares methods for policy evaluation that are efficient in the amount of data used (Bradtke & Barto 1996; Boyan 1999, 2002; Geramifard, Bowling & Sutton 2006).”
Emphatic Temporal-Difference Learning (2015, methodology): “Keywords: temporal-difference learning, function approximation, off-policy learning, stability, convergence”
Model-Free Preference-Based Reinforcement Learning (2016, methodology): “The LSTD regularization factors, the RBF bandwith and number of basis function, are only tuned once per domain, to determine a reasonable setup.”
Least Squares Temporal Difference Methods: An Analysis under General Conditions (2012, methodology): “It was proposed in Bertsekas and Yu [8], and it extends the least squares temporal diﬀerence (LSTD) algorithm for on-policy learning (Bradtke and Barto [12] and Boyan [11]).”
A Kernel Loss for Solving the Bellman Equation (2019, background): “either on- and off-policy cases, can be found by various algorithms (e.g., Sutton, 1988; Boyan, 1999; Sutton et al., 2009; Dann et al., 2014), and its theoretical properties have been extensively studied (e.”
Reinforcement Learning for Humanoid Robotics (2003, methodology): “Using this set of simultaneous linear equations, a solution to Equation (27) can be obtained by adapting the LSTD(λ) policy evaluation algorithm [5, 7, 12].”
Reinforcement learning for sensing strategies (2004, methodology): “In our case, we use a mixed discrete / continuous representation of the state space.”
Learning Robust Policies for Object Manipulation with Robot Swarms (2018, methodology): “To define the feature function for LSTD, we further select 1000 samples randomly from the SARS data set just as another 500 samples as sparse subset for the GP. Figure 3 depicts the value function after 15 iterations for three different translation-rotation ratios ρ.”
Optimality and Approximation with Policy Gradient Methods in Markov Decision Processes (2019, methodology): “In contrast Peters and Schaal [2008] utilize Least Squares Temporal Difference methods [Boyan, 1999] to minimize the loss.”
A unifying framework for computational reinforcement learning theory (2009, methodology): “Right: ExpressWorld (adapted from Boyan and Moore [1995]).”