Basis Function Adaptation in Temporal Difference Reinforcement Learning

By Ishai Menache, Shie Mannor, Nahum Shimkin. 2005

208 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

State Representation Learning in Robotics: Using Prior Knowledge about Physical Interaction (2014, background): “[20] also adapt the state representation during reinforcement learning; they represent the state as a set of basis functions and adapt their parameters in order to improve the value function estimate.”
Algorithms for Reinforcement Learning (2010, methodology): “These include tuning the parameter of Gaussian RBF either using a gradient- or the cross-entropymethod in the context of LSTD (Menache et al., 2005), deriving new basis functions with nonparametric techniques (Keller et al., 2006; Parr et al., 2007) or using a combination of numerical analysis and…”
Basis Adaptation for Sparse Nonlinear Reinforcement Learning (2013, methodology): “…Mannor 2005) enables tuning a given parametric basis, such as the Fourier basis (Konidaris, Osentoski, and Thomas 2011), radial basis functions (RBF) (Menache, Shimkin, and Mannor 2005), polynomial bases (Lagoudakis and Parr 2003), etc. to the geometry of a particular Markov decision process (MDP).”
Regularization and feature selection in least-squares temporal difference learning (2009, methodology): “Another class of methods that bear some similarity to our own are recent methods for feature generation based on the Bellman error (Menache et al., 2005; Keller et al., 2006; Parr et al., 2007).”
A Tutorial on the Cross-Entropy Method (2005, background): “... models (Homem-de-Mello and Rubinstein, 2002); queueing models of telecommunication systems (de Boer, 2000; de Boer, Kroese, and Rubinstein, 2004); neural computation (Dubin, 2002, 2004); control and navigation (Helvik and Wittner, 2001); DNA sequence alignment (Keith and Kroese, 2002); schedulin”
Learning state representations with robotic priors (2015, background): “Menache et al. [25] also adapt the state representation during reinforcement learning; they represent the state as a set of basis functions and adapt their parameters in order to improve the value function estimate.”
Proto-value Functions: A Laplacian Framework for Learning Representation and Control in Markov Decision Processes (2007, background): “This differs fundamentally from many past attempts at basis function generation, for example tuning the parameters of pre-defined basis functions (Menache et al., 2005; Kveton and Hauskrecht, 2006), dynamically allocating new parametric basis functions based on state space trajectories (Kretchmar…”
Policy evaluation with temporal differences: a survey and comparison (2015, methodology): “In contrast, Menache et al. (2005) assumes a fixed class of features, for example, RBFs, and optimizes only the free parameters (e.g., the basis function widths) by gradient descent or by using the crossentropy optimization method (De Boer et al., 2010).”
Automatic basis function construction for approximate dynamic programming and reinforcement learning (2006, methodology): “Recently, a significant amount of work has been devoted to constructing bases for value functions automatically, e.g., (Mahadevan, 2005; Mannor et al., 2005; Ratitch & Precup, 2004; Smart, 2004; Ziv & Shimkin, 2005).”
Artificial curiosity with planning for autonomous perceptual and cognitive development (2011, background): “are many works [12], [7] that develop perceptual systems in this way.”
Control Techniques for Complex Networks: Workload (2007, methodology): “Methods to constructa basis{ψi} based on observations of a Markov model are described in [474, 350].”
A unifying framework for computational reinforcement learning theory (2009, methodology): “In the context of MDP planning and learning, interesting progress has been made recently for feature section, including the use of Bellman errors [Menache et al. 2005; Keller et al. 2006; Parr et al. 2007], spectral analysis [Mahadevan and Maggioni 2007], Lasso-type regularization [Loth et al.”
Approximate dynamic programming via iterated Bellman inequalities (2015, background): “This is large topic and an active area of research; we direct the interested reader to [38, 39, 40, 41, 42, 43, 44, 45] and the references therein.”