Analyzing feature generation for value-function approximation

By Ronald E. Parr, Christopher Painter-Wakefield, Lihong Li, Michael L. Littman. 2007

156 citations recorded by the source corpus.

Subfields: Control and Systems Engineering, Statistical and Nonlinear Physics, Reinforcement Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Statistical and Nonlinear Physics

In Reinforcement Learning

In Signal Processing

Citations in the corpus

Investigating Contingency Awareness Using Atari 2600 Games (2012, methodology): “While automatic feature construction methods (Parr et al. 2007; Konidaris, Osentoski, and Thomas 2011) and generic feature sets (Schweitzer and Seidmann 1985) have been proposed, current techniques do not explicitly incorporate any notion of contingency awareness .”
Fitted Q-Learning for Relational Domains (2020, methodology): “Value function approximation in Reinforcement Learning (RL) has long been viewed using the lens of feature discovery (Parr et al. 2007).”
Successor Features for Transfer in Reinforcement Learning (2016, methodology): “When it comes to representation, a lot of effort in previous research has been directed towards the development of methods to automatically compute good features to represent the value function [14, 10, 16, 15].”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, methodology): “9 using the BEBF basis [Parr et al. , 2007], and Figure 2 shows the value function approximation result, where the cyan curve”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “The approaches of Parr et al. (2007), Painter-Wakefield and Parr (2012a) and Geramifard et al. (2013, 2011) are based on the orthogonal matching principle (Pati et al., 1993) and incrementally add features which have high correlation with the temporal-difference error.”
An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning (2008, methodology): “Some initial results on the effects of noise in feature generation for BEBF/MEBF/Krylov methods can be found in Parr et al. (2007) , however further analysis would still be helpful.”
Evolutionary Computation for Reinforcement Learning (2012, methodology): “In fact, though methods exist for automatically discovering good representations for value-functions (Mahadevan and Maggioni , 2007; Parr et al, 2007) temporal-difference methods typically also use fixed representations for function approximation.”
Value-Aware Loss Function for Model-based Reinforcement Learning (2017, background): “…(e.g., [Silver et al., 2007; Liang et al., 2016]) or automatically generated data-dependently, e.g., [Petrik, 2007; Mahadevan and Maggioni, 2007; Parr et al., 2007; Mahadevan and Liu, 2010; Geramifard et al., 2011; Farahmand and Precup, 2012; B¨ohmer et al., 2013; Milani Fard et al., 2013].”
Autonomous Learning of State Representations for Control: An Emerging Field Aims to Autonomously Learn State Representations for Reinforcement Learning Agents from Their Real-World Sensor Observations (2015, background): “The first type aims to represent the propagated reward (Krylov-bases [41], BEBF [40] and BARB [33]).”
Construction of approximation spaces for reinforcement learning (2013, methodology): “In difference to the classical analysis put forward by Petrik (2007), Parr et al. (2007) and Mahadevan and Maggioni (2007), this point of view puts emphasis on the reuse of prior experience, as investigated in transfer learning (Taylor and Stone, 2009; Ferguson and Mahadevan, 2006; Ferrante et al.,…”
Regularization in reinforcement learning (2011, methodology): “Another basis generation approach is to use the Bellman residual for defining new basis functions [Parr et al., 2007].”
RLPy: a value-function-based reinforcement learning framework for education and research (2015, methodology): “powerful framework and aim at improving existing methods by, for example, feature learning (Keller et al., 2006; Parr et al., 2007; Geramifard et al., 2011), or policies with better exploration-exploitation trade-off (Nouri and Littman, 2009; Jaksch et al.”
A unifying framework for computational reinforcement learning theory (2009, methodology): “In the context of MDP planning and learning, interesting progress has been made recently for feature section, including the use of Bellman errors [Menache et al. 2005; Keller et al. 2006; Parr et al. 2007], spectral analysis [Mahadevan and Maggioni 2007], Lasso-type regularization [Loth et al.”
Regularization and feature selection in least-squares temporal difference learning (2009, methodology): “In particular (Parr et al., 2007) analyze approaches that continually add new basis functions to the current set, based upon their correlation with the Bellman residual.”
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes (2007, methodology): “…dynamically allocating new parametric basis functions based on state space trajectories (Kretchmar and Anderson, 1999), or generating basis functions using the Bellman error in approximating a specific value function (Keller et al., 2006; Petrik, 2007; Parr et al., 2007; Patrascu et al., 2002).”
Basis Function Discovery Using Spectral Clustering and Bisimulation Metrics (2011, background): “The second category of methods aims to construct basis functions that reduce the error in value function estimation (also known as the Bellman error), e.g. (Keller, Mannor , and Precup 2006; Parr et al. 2008).”
Representation Discovery for MDPs Using Bisimulation Metrics (2015, methodology): “Several methods have been developed to obtain accurate representations for a given policy π(Parr et al. 2008b; 2008a).”
Incremental Basis Construction from Temporal Difference Error (2011, methodology): “…make use of the so-called Bellman error basis functions (BEBFs; Wu and Givan (2005); Keller et al. (2006); Parr et al. (2007); Mahadevan and Liu (2010)), which capture the intuition that the " Bellman error, loosely speaking, point[s] towards the optimal value function " (Parr et al., 2007).”
Online Discovery of Feature Dependencies (2011, background): “While batch AFAs (Keller et al., 2006; Mahadevan et al., 2006; Parr et al., 2007) are promising, their computational demand often limit their application in online settings.”