An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning

By Ronald E. Parr, Lihong Li, Gavin Taylor, Christopher Painter-Wakefield, Michael L. Littman. 2008

224 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

When to use parametric models in reinforcement learning? (2019, methodology): “In fact, if we fit a linear model to the data and then fully solve this model, the solution is equal to the LSTD solution [Parr et al., 2008].”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “A similar connection between the quality of representations and model based objectives in the linear setting was made by Parr et al. (2008).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “In the linear function approximation case, it is known that modelbased updates and fitted value iteration updates actually produce identical iterates (Vanseijen and Sutton, 2015; Parr et al., 2008), though it is unknown whether this relationship holds under non-linear function approximation.”
Modelling transition dynamics in MDPs with RKHS embeddings (2012, background): “Finally, in (Parr et al., 2008) a way is proposed to approximate expectations in a low dimensional state representation.”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, methodology): “The least-squares estimator at the last stage, β̂T , minimizes the sum of squared errors between the Q-value at time T , QT = RT , and the fitted Q-function, Q̂T , over the training data (Lagoudakis and Parr 2003; Parr et al. 2008):”
Temporal Difference Models: Model-Free Deep RL for Model-Based Control (2018, methodology): “We are not the ﬁrst to study the connection between model-free and model-based methods, with Boyan (1999) and Parr et al. (2008) being two notable examples.”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “In this case, the TD solution coincides with that of a model-based approach (Parr et al., 2008) known as certainty equivalence (Kumar & Varaiya, 1986).”
Agnostic System Identification for Monte Carlo Planning (2015, background): “Schoknecht (2002), Parr et al. (2008), and Sutton et al. (2008) all point out the equivalence between the optimal value function in an approximate linear model and an approximate linear value function learned without a model.”
Algorithms for Reinforcement Learning (2010, background): “Since, by assumption, the features are linearly independent, the minimizer (8)Parr et al. (2008) observed this for λ = 0.”
Reinforcement learning with misspecified model classes (2013, methodology): “Model-free Value Based (VB) techniques sidestep learning an explicit world model and directly compute a value function, although [26] showed that MB and VB RL methods using linear representations are equivalent.”
A Deeper Look at Planning as Learning from Replay (2015, background): “The distinction between model-free and model-based methods is further blurred by the theoretical result that the fixed point of a linear, least-squares model on a set of samples is the same as the LSTD solution of those samples (Sutton et al., 2008; Parr et al., 2008).”
Model-Based Reinforcement Learning with Value-Targeted Regression (2020, background): “Moving beyond tabular MDP, there have been signicant theoretical and empirical advances on RL with function approximation, including but not limited to Baird (1995); Tsitsiklis & Van Roy (1997); Parr et al. (2008); Mnih et al. (2013, 2015); Silver et al. (2017); Yang & Wang (2019b); Bradtke ”
Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound (2019, methodology): “For instance (Baird, 1995; Tsitsiklis and Van Roy, 1997; Parr et al., 2008; Mnih et al., 2013, 2015; Silver et al., 2017; Yang and Wang, 2019).”
Strengths, Weaknesses, and Combinations of Model-based and Model-free Reinforcement Learning (2015, background): “It has been shown that prediction with the linear model leads to an asymptotic solution equivalent to the linear TD(0) solution (Parr et al., 2008; Sutton et al., 2008).”
Sample-Optimal Parametric Q-Learning Using Linearly Additive Features (2019, methodology): “A similar argument has been made in Parr et al. [2008].”
Kernelized value function approximation for reinforcement learning (2009, methodology): “In the linear approximation case that the Bellman error can be broken down into components that express the approximation s error in estimating the process s reward and transition functions (Parr et al., 2008).”