Regularization and feature selection in least-squares temporal difference learning

By J. Kolter, Andrew Y. Ng. 2009

232 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Reinforcement learning in robotics: A survey (2013, background): “…flight (Bagnell and Schneider, 2001), learning biped walk patterns (Tedrake, 2004; Tedrake et al., 2005), driving a radiocontrolled (RC) car, learning a jumping behavior for a robot dog (Kolter and Ng, 2009a) (illustrated in Figure 5), and balancing a two-wheeled robot (Vlassis et al., 2009).”
Action-Gap Phenomenon in Reinforcement Learning (2011, methodology): “…can be any online or ofﬂine, batch or incremental algorithms of choice such as Q-learning, SARSA [15], and their variants [17], LSPI [1], LARS-TD [18] in a policy iteration procedure, REG-LSPI [13], various Fitted Q-Iterations algorithms [19, 20, 12], or Linear Programming-based approaches [21,…”
Group-driven Reinforcement Learning for Personalized mHealth Intervention (2017, background): “The least-square temporal diﬀerence for Q-value (LSTD Q ) [8,7] provides a closed-form solver for (3) as follows Since where Q ( s i , a ; (cid:98) w ) = x ( s i , a ) (cid:98) w is the newly estmated Q-value via the critic updating (4) and ζ a is the tuning parameter to prevent overﬁtting 2 .”
Sparse Q-learning with Mirror Descent (2012, methodology): “proximation theory [Bor08]. l 1 regularized methods for solving MDPs have become a topic of recent attention, including a technique combining least-squares TD (LSTD) with least-angle regression (LARS)[KN09]; anothermethodforcombiningl 1 regularization with approximate linear programming [PTPZ10]; ﬁn”
Scalable Bilinear π Learning Using State and Action Features (2018, background): “Last but not least, how can we inject prior knowledge, such as sparsity (Kolter and Ng, 2009), in the primal-dual formulation and develop corresponding algorithms to take advantage of such information?”
Sparse multi-task reinforcement learning (2014, unknown): “In RL the idea of sparsity has been successfully integrated into policy evaluation [9, 13, 15, 23] but rarely in the full policy iteration.”
Robot learning from demonstration by constructing skill trees (2012, methodology): “If necessary, abstraction selection could be paired with a feature selection method (Kolter and Ng 2009; Johns et al. 2010) to augment the selected abstraction during policy learning.”
Learning from Limited Demonstrations (2013, methodology): “Most methods attempt to exploit the structures in the value function [8, 9, 10, 11], but in some problems one might have extra information about the structure of good or optimal policies as well.”
Incremental Basis Construction from Temporal Difference Error (2011, background): “In addition, basis function selection under sparsity constraints (Kolter and Ng, 2009) can also be seen as a special case where the W are sparse binary matrices with each wn containing exactly one non-zero entry.”
Regularized Off-Policy TD-Learning (2012, methodology): “To make a fair comparison, we use the same basis function setting as in [7], where two dimensional grids of 2, 4, 8, 16, 32 RBFs are used so that there are totally 1365 basis functions.”
Statistical linear estimation with penalized estimators: an application to reinforcement learning (2012, background): “…(Bickel et al., 2009; Koltchinskii, 2011; Bühlmann and Geer, 2011) and the references therein), as well as in the reinforcement learning setting (Kolter and Ng, 2009; Ghavamzadeh et al., 2010; 2011; Maillard, 2011), mainly because it allows for non-trivial performance bounds even when the…”
Improved empirical methods in reinforcement-learning evaluation (2015, methodology): “For example, we could have the modular learner be a selective learner by using LARS-TD (Kolter and Ng, 2009).”
Error Propagation for Approximate Policy and Value Iteration (2010, unknown): “Kolter and Ng [10] formulated an l1 regularization extension of LSTD.”
Generalization and Regularization in DQN (2018, background): “The few existing studies often focus on single-task settings with linear function approximation (e.g., Farahmand et al., 2008; Kolter and Ng, 2009).”
Basis Adaptation for Sparse Nonlinear Reinforcement Learning (2013, methodology): “Basis selection methods combine sparse feature selection through L1 regularization with traditional least-squares type RL methods (Kolter and Ng 2009; Johns, Painter-Wakefield, and Parr 2010), linear complementarity methods (Johns, Painter-Wakefield, and Parr 2010), approximate linear programming (P”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, background): “2 shows that mirror-descent TD converges more quickly with far smaller Bellman errors than LARS-TD [68] on a discrete “tworoom” MDP [69].”
A Non-Parametric Approach to Dynamic Programming (2011, background): “Regularization has a similar effect to changing the transition probabilities of the system [25].”