Regularized Off-Policy TD-Learning

By Bo Liu, Sridhar Mahadevan, Ji Liu. 2012

65 citations recorded by the source corpus.

We present a novel l1 regularized off-policy convergent TD-learning method (termed RO-TD), which is able to learn sparse representations of value functions with low computational complexity. The algorithmic framework underlying RO-TD integrates two key ideas: off-policy convergent gradient TD methods, such as TDC, and a convex-concave saddle-point formulation of non-smooth convex optimization, which enables first-order solvers and feature selection using online convex regularization. A detailed theoretical and experimental analysis of RO-TD is presented. A variety of experiments are presented to illustrate the off-policy convergence, sparse feature selection capability and low computational cost of the RO-TD algorithm.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

Citations in the corpus

Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning (2016, background): “In the literatures of reinforcement learning, there have been works on dual temporal diﬀerence learning which use primal-dual-type methods, see for examples (Yang et al., 2009; Liu et al., 2012; Mahadevan and Liu, 2012; Mahadevan et al., 2014).”
High-Confidence Off-Policy Evaluation (2015, methodology): “Existing methods for estimating the performance of the evaluation policy using trajectories from behavior policies do not provide confidence bounds (Maei and Sutton 2010; Liu, Mahadevan, and Liu 2012; Mandel et al. 2014).”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, methodology): “…provide different approaches to formulating the problem, ﬁrst as a variational inequality problem [Juditsky et al. , 2008; Mahadevan et al. , 2014] or as a linear inverse problem [Liu et al. , 2012], or as a quadratic objective function (MSPBE) using two-time-scale solvers [Qin and Li, 2014].”
An online primal-dual method for discounted Markov decision processes (2016, background): “In the RL literatures, there have been some works about dual temporal difference learning and primaldual methods, see for examples [13], [15], [16], [24].”
Basis Adaptation for Sparse Nonlinear Reinforcement Learning (2013, unknown): “…(Kolter and Ng 2009; Johns, Painter-Wakefield, and Parr 2010), linear complementarity methods (Johns, Painter-Wakefield, and Parr 2010), approximate linear programming (Petrik et al. 2010), or convex-concave optimization methods for sparse off-policy TD-learning (Liu, Mahadevan, and Liu 2012).”
Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize (2015, background): “In the gradient-TD algorithms [47, 45, 23] and the proximal gradient-TD algorithms [22, 24] (see also [21, 25]), the difficulty in TD(λ) is overcome by reformulating the approximate policy evaluation problem TD(λ) attempts to solve as optimization problems and then tackle them with optimization tech”
Policy evaluation with temporal differences: a survey and comparison (2015, background): “, 2012) and reformulating the regularized objective as a convex-concave saddle-point problem (Liu et al., 2012).”
Regularized Policy Iteration with Nonparametric Function Spaces (2016, methodology): “Another possibility is to use stochastic gradient-like algorithms similar to the works of Liu et al. (2012); Qin et al. (2014).”
Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spaces (2014, background): “Nash games are closely related to saddle point problems [129, 130, 135].”