Least-Squares Policy Iteration

By Michail G. Lagoudakis, Ronald E. Parr. 2003

1,500 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “Since in RL we do not have access internal expectation, we instead minimize the Bellman residual [43][28, 3] which has been extensively used in DRL literature [36, 47]:”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, background): “We next turn to the task of controlling the classical inverted pendulum system [18, 19] from images.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “In practice, least-squares fixed-point iteration can give rise to more effective policies as compared to the Bellman residual, while the Bellman residual minimization approach can be more stable and predictable (Lagoudakis and Parr, 2003).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “[29] Michail G Lagoudakis and Ronald Parr.”
Deep learning in neural networks: An overview (2014, background): “The contents of this paper may be used for educational and noncommercial purposes, including articles for Wikipedia and similar sites.”
Fitted Q-Learning for Relational Domains (2020, methodology): “…Difference (LSTD) Learning (Boyan 1999), Least Squares Policy Evaluation (LSPE) (Yu and Bertsekas 2009) and Least Sqaure Policy iteration (LSPI) (Lagoudakis and Parr 2003) approximate the Q-function as linear function of the parameter vector i.e: ˆ Q ( s, a ) = w · φ where w is the parameter of…”
Reinforcement Learning in Healthcare: A Survey (2019, methodology): “inear approximator cases of Neural Fitted Q Iteration (NFQI [40]), the Tree-based FQI [41], and robust linear approximation techniques for policy learning such as Least-Squares Policy Iteration (LSPI [42]). As will be discovered later, these BRL methods have enjoyed wide and successful applications ”
Modelling transition dynamics in MDPs with RKHS embeddings (2012, methodology): “We compared our approach in policy learning to the GP based LSPI approach (1st and 4th plot) and in value learning to NPDP (2nd and 3rd plot).”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “…programming (Bertsekas & Tsitsiklis, 1996; Munos, 2005) and approximate policy iteration (Farahmand et al., 2010; Scherrer et al., 2015); for the latter of which there exist well known algorithms that are stable at least with linear function approximation (see e.g. Lagoudakis & Parr (2003)).”
Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, methodology): “For example, when both importance weights and value functions are modeled using the same linear class, we recover LSTDQ (Lagoudakis and Parr, 2004) and off-policy LSTD (Bertsekas and Yu, 2009; Dann et al., 2014) as special cases of MWL/MQL and their state-function variants.”
Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning (2019, methodology): “ased approach (Sutton and Barto, 2018). When the sample space and action space are continuous, we can apply some functional approximation to q-function modeling and use the temporal-dierence method (Lagoudakis and Parr, 2004). Once we have an estimate q^, the DM estimate is simply ˆ^ DM = (1 ) P N[E”
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation (2018, background): “However, our idea can be extended to estimating value functions as well, by using estimated density ratios to weight observed transitions (c.f., the distribution in LSTDQ [16]).”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “The LSTD(λ) method was always relatively robust to off-policy training (e.g., Lagoudakis & Parr 2003, Yu 2010, Mahmood, van Hasselt & Sutton 2014), but its per-step computational complexity is quadratic in the number of parameters of the function approximator, as opposed to the linear complexity of…”
Linear Off-Policy Actor-Critic (2012, methodology): “Least-squares methods such as LSTD (Bradtke & Barto, 1996) and LSPI (Lagoudakis & Parr, 2003) can be used oﬀ-policy and are sound with linear function approximation, but are computationally expensive; their complexity scales quadratically with the number of features and weights.”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, methodology): “The least-squares estimator at the last stage, β̂T , minimizes the sum of squared errors between the Q-value at time T , QT = RT , and the fitted Q-function, Q̂T , over the training data (Lagoudakis and Parr 2003; Parr et al. 2008):”
Approximate Policy Iteration Schemes: A Comparison (2014, methodology): “It is natural to also consider the algorithm API( α ) (mentioned by Lagoudakis & Parr (2003a)), a variation of API that is conservative like CPI( α ) in the sense that it mixes the new policy with the previous ones with weights α and 1 − α , but that directly uses the distribution ν in the…”
Regularized Behavior Value Estimation (2021, methodology): “In particular, given that policy iteration algorithms typically do not require more than a few steps to converge to optimal policy (Lagoudakis and Parr, 2003; Sutton and Barto, 2018, Chapter 4.3), we focus on the extreme scenario where we are allowed to take a single improvement step.”