Basis Adaptation for Sparse Nonlinear Reinforcement Learning

By Sridhar Mahadevan, Stephen Giguere, Nicholas Jacek. 2013

11 citations recorded by the source corpus.

This paper presents a new approach to representation discovery in reinforcement learning (RL) using basis adaptation. We introduce a general framework for basis adaptation as {\em nonlinear separable least-squares value function approximation} based on finding Frechet gradients of an error function using variable projection functionals. We then present a scalable proximal gradient-based approach for basis adaptation using the recently proposed mirror-descent framework for RL. Unlike traditional temporal-difference (TD) methods for RL, mirror descent based RL methods undertake proximal gradient updates of weights in a dual space, which is linked together with the primal space using a Legendre transform involving the gradient of a strongly convex function. Mirror descent RL can be viewed as a proximal TD algorithm using Bregman divergence as the distance generating function. We present a new class of regularized proximal-gradient based TD methods, which combine feature selection through sparse L1 regularization and basis adaptation. Experimental results are provided to illustrate and validate the approach.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistical and Nonlinear Physics

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, background): “More advanced gradient methods exist, such as those based on natural gradients [9] and mirror descent algorithms [45], which often allow for faster convergence to near-optimal policies.”