Least Squares Policy Evaluation Algorithms with Linear Function Approximation

By Angelia Nedić, Dimitri P. Bertsekas. 2003

229 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Networks and Communications, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Statistics and Probability

Citations in the corpus

Off-policy learning with eligibility traces: a survey (2013, background): “The first one is when one considers on-policy learning, as Nedić and Bertsekas (2003) did when they derived the first convergence proof of (on-policy) LSPE(λ).”
Least Squares Temporal Difference Methods: An Analysis under General Conditions (2012, neutral): “11”
Algorithms for Reinforcement Learning (2010, background): “Under standard assumptions on the sample and when nt = t, λ-LSPE is known to converge almost surely to the solution of the projected fixed-point equation (22), both for decreasing (Nedič and Bertsekas, 2003) and constant step-sizes (Bertsekas et al., 2004).”
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes (2007, methodology): “However, most approaches (with notable exceptions discussed below) are based on a fixed parametric architecture, and a parameter estimation method is used to approximate value functions, such as temporal-difference learning (Sutton and Barto, 1998; Tsitsiklis and Van Roy, 1997), least squares projec”
Scalable Bilinear π Learning Using State and Action Features (2018, methodology): “Linear basis functions have been used for both policy evaluation and optimization (Tsitsiklis and Van Roy, 1997; Nedić and Bertsekas, 2003; Lagoudakis and Parr, 2003; Melo et al., 2008; Sutton et al., 2009).”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, methodology): “Although many more sophisticated methods have been developed over the past three decades, such as least-squares based temporal difference approaches, including LSTD [23], LSPE [24] and LSPI [25], first-order temporal difference learning algorithms may scale more gracefully to high dimensional proble”
Stabilization of Stochastic Iterative Methods for Singular and Nearly Singular Linear Systems (2014, methodology): “Also, the sample sequence can be generated through state transitions of an irreducible Markov chain as, for example, in temporal difference methods for cost evaluation problems in ADP (see Bradtke and Barto [12], Boyan [11], Nedić and Bertsekas [40], Bertsekas [4]) or for general projected equations”
Dynamic modeling and control of supply chain systems: A review (2008, background): “Patrinos and Sarimveis [177] developed an optimistic variant of policy iteration coupled with least-squares policy evaluation [178].”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “A projection matrix as defined in (8) is used in the analysis and in the derivation of several algorithms (Tsitsiklis and Van Roy, 1997; Nedić and Bertsekas, 2003; Bertsekas et al, 2004; Sutton et al, 2008, 2009; Maei and Sutton, 2010).”
Full Paper Reinforcement Learning for Imitating Constrained Reaching Movements (methodology): “For the approximation, an adaptation of a LSTD ( λ ) (Least Square Temporal Diﬀerence) algorithm [9, 17] [25] is used (see Appendix).”
Least-Squares Policy Iteration (2003, background): “The convergence properties of the algorithm are not affected by this change (Nedić and Bertsekas, 2003).”
Policy Gradient Methods for Robot Control (2003, methodology): “In future work, we hope to use the estimate of the previous gradients to initialize our subsequent rollouts using the results given in [12] – basically, by using LSTD-like algorithms which can incorporate an initial guess for the parameters and still be unbiased.”