Automatic basis function construction for approximate dynamic programming and reinforcement learning

By Philipp W. Keller, Shie Mannor, Doina Precup. 2006

183 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

Citations in the corpus

Manifold Embeddings for Model-Based Reinforcement Learning under Partial Observability (2009, background): “Approaches for efficient function approximation, basis function construction, and discovery of embeddings has been the topic of significant investigations [3, 11, 20, 15, 13].”
Artificial curiosity with planning for autonomous perceptual and cognitive development (2011, background): “are many works [12], [7] that develop perceptual systems in this way.”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, methodology): “As a result, meth-ods in representation learning have ﬂourished, ranging from basis adaptation (Menache et al., 2005; Keller et al., 2006), proto-value functions (Mahadevan and Maggioni, 2007), gradient-based learning (Yu and Bertsekas, 2009), and feature generation schemes such as tile coding…”
Basis Function Discovery Using Spectral Clustering and Bisimulation Metrics (2011, background): “The second category of methods aims to construct basis functions that reduce the error in value function estimation (also known as the Bellman error), e.g. (Keller, Mannor , and Precup 2006; Parr et al. 2008).”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “Keller et al. (2006) uses neighborhood component analysis, a dimensionality reduction techniques for labeled data, to project the high-dimensional state space to a lower dimensional feature representation.”
Linear Fitted-Q Iteration with Multiple Reward Functions (2012, background): “However, the question of how s t should be defined can be answered at least in part by the data themselves together with expert knowledge and feature/model selection techniques analogous to those used in supervised learning settings (Keller et al., 2006) if we have an adequate definition of r t .”
Pose-Sensitive Embedding by Nonlinear NCA Regression (2010, methodology): “NCA has also been recently extended to the nonlinear case [34] using MNIST class labels and to linear 1D regression for reinforcement learning [20].”
Algorithms for Reinforcement Learning (2010, methodology): “nonparametric techniques (Keller et al., 2006; Parr et al., 2007) or using a combination of numerical analysis and nonparametric techniques (Mahadevan, 2009).”
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes (2007, methodology): “…dynamically allocating new parametric basis functions based on state space trajectories (Kretchmar and Anderson, 1999), or generating basis functions using the Bellman error in approximating a specific value function (Keller et al., 2006; Petrik, 2007; Parr et al., 2007; Patrascu et al., 2002).”
Analyzing feature generation for value-function approximation (2007, methodology): “This question is asked both in the context of graph-based approaches (Mahadevan & Maggioni, 2006) and Bellman-error-based methods (Keller et al., 2006).”
RLPy: a value-function-based reinforcement learning framework for education and research (2015, methodology): “powerful framework and aim at improving existing methods by, for example, feature learning (Keller et al., 2006; Parr et al., 2007; Geramifard et al., 2011), or policies with better exploration-exploitation trade-off (Nouri and Littman, 2009; Jaksch et al.”
Incremental Basis Construction from Temporal Difference Error (2011, methodology): “A significant fraction of the existing approaches make use of the so-called Bellman error basis functions (BEBFs; Wu and Givan (2005); Keller et al. (2006); Parr et al. (2007); Mahadevan and Liu (2010)), which capture the intuition that the “Bellman error, loosely speaking, point[s] towards the…”
A unifying framework for computational reinforcement learning theory (2009, methodology): “In the context of MDP planning and learning, interesting progress has been made recently for feature section, including the use of Bellman errors [Menache et al. 2005; Keller et al. 2006; Parr et al. 2007], spectral analysis [Mahadevan and Maggioni 2007], Lasso-type regularization [Loth et al.”
Approximate dynamic programming via iterated Bellman inequalities (2015, background): “This is large topic and an active area of research; we direct the interested reader to [38, 39, 40, 41, 42, 43, 44, 45] and the references therein.”
Online Discovery of Feature Dependencies (2011, background): “While batch AFAs (Keller et al., 2006; Mahadevan et al., 2006; Parr et al., 2007) are promising, their computational demand often limit their application in online settings.”
An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning (2008, background): “Many researchers have proposed using features based upon the residual error in the current feature set (Wu & Givan, 2004; Sanner & Boutilier, 2005; Keller et al., 2006). Parr et al. (2007) describe this family of techniques as Bellman Error Basis Functions (BEBFs), and analyze some of the properties”
Regularization and feature selection in least-squares temporal difference learning (2009, methodology): “Another class of methods that bear some similarity to our own are recent methods for feature generation based on the Bellman error (Menache et al., 2005; Keller et al., 2006; Parr et al., 2007).”