Statistical linear estimation with penalized estimators: an application to reinforcement learning

By B. '. Pires, Csaba Szepesvari. 2012

34 citations recorded by the source corpus.

Motivated by value function estimation in reinforcement learning, we study statistical linear inverse problems, i.e., problems where the coefficients of a linear system to be solved are observed in noise. We consider penalized estimators, where performance is evaluated using a matrix-weighted two-norm of the defect of the estimator measured with respect to the true, unknown coefficients. Two objective functions are considered depending whether the error of the defect measured with respect to the noisy coefficients is squared or unsquared. We propose simple, yet novel and theoretically well-founded data-dependent choices for the regularization parameters for both cases that avoid data-splitting. A distinguishing feature of our analysis is that we derive deterministic error bounds in terms of the error of the coefficients, thus allowing the complete separation of the analysis of the stochastic properties of these errors. We show that our results lead to new insights and bounds for linear value function estimation in reinforcement learning.

Subfields: Artificial Intelligence, Biomedical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

Citations in the corpus

Information-Theoretic Considerations in Batch Reinforcement Learning (2019, positive): “While results similar to Theorems 2 and 11 exist (Farahmand 2011, Chapter 5; see also Lazaric et al. (2012); Pires & Szepesv´ari (2012); Farahmand et al. (2016)), according to our knowledge, fast rate for the minimax al-gorithm has not been established before: for example, Antos et al. (2008);…”
Finite-Sample Analysis of Proximal Gradient TD Algorithms (2015, background): “A saddle point formulation for off-policy TD learning was initially explored in Liu et al. [2012], where the objective function is the norm of the approximation residual of a linear inverse problem [ ´Avila Pires and Szepesv´ari, 2012].”
Regularized Policy Iteration with Nonparametric Function Spaces (2016, methodology): “An example of such an approach for parametric spaces has been analyzed by Ávila Pires and Szepesvári (2012).”
Weak Convergence Properties of Constrained Emphatic Temporal-difference Learning with Constant and Slowly Diminishing Stepsize (2015, background): “It can also be applied to finding approximate solutions under additional penalty terms suggested by [33].”
A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximation (2018, background): “First, among others, Antos et al. [2008], Lazaric et al. [2010], Ghavamzadeh et al. [2010], Pires and Szepesv´ari [2012], Prashanth et al. [2013] and Tu and Recht [2018] analyze least-squares temporal difference learning (LSTD).”
Bellman-consistent Pessimism for Offline Reinforcement Learning (2021, unknown): “The on-support part matches the n rate (fast rate error bound) of API or AVI analysis (e.g., [Pires and Szepesvári, 2012; Lazaric et al., 2012; Chen and Jiang, 2019]).”
Provably Good Batch Reinforcement Learning Without Great Exploration (2020, result): “We make a few remarks about the comparison between this result and prior related results below: Remark 1 (Comparison with prior API/AVI bounds).”