Data-driven calibration of linear estimators with minimal penalties

By Sylvain Arlot, Francis R. Bach. 2009

61 citations recorded by the source corpus.

This paper tackles the problem of selecting among several linear estimators in non-parametric regression; this includes model selection for linear regression, the choice of a regularization parameter in kernel ridge regression or spline smoothing, and the choice of a kernel in multiple kernel learning. We propose a new algorithm which first estimates consistently the variance of the noise, based upon the concept of minimal penalty which was previously introduced in the context of model selection. Then, plugging our variance estimate in Mallows' CL penalty is proved to lead to an algorithm satisfying an oracle inequality. Simulation experiments with kernel ridge regression and multiple kernel learning show that the proposed algorithm often improves significantly existing calibration procedures such as 10-fold cross-validation or generalized cross-validation.

Subfields: Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Sharp analysis of low-rank kernel matrix approximations (2012, neutral): “(13) in Appendix B for details), doing so with respect to ε would require additional assumptions, which are standard in the analysis in ridge regression (Hsu et al., 2011; Arlot and Bach, 2009), but that would make the results signiﬁcantly more complicated.”
High-dimensional regression with unknown variance (2011, background): “Arlot and Bach [4] generalize the slope heuristic to symmetric linear estimators with spectrum in [0,1] and prove an oracle bound for the resulting estimator.”
Rejoinder on: Minimal penalties and the slope heuristics: a survey. (2019, unknown): “Third, the key deviation bound proved by Bellec and Yang (2019) includes a non-negative term c k ( A λ − A ∗ ) k 2 , while generalizing the results of Arlot and Bach (2011) would require a similar uniform deviation bound without this term.”
Union Support Recovery in Multi-task Learning (2010, unknown): “For recent work on data-driven tuning of the penalty parameters, we refer the reader to Arlot and Bach (2009).”
Sharp Oracle Inequalities for Aggregation of Affine Estimators (2011, methodology): “It is worth noting that oracle inequalities for penalized empirical risk minimizer were also proved by Golubev [35], and for model selection by Arlot and Bach [3], Baraud, Giraud and Huet [5].”
Self-concordant analysis for logistic regression (2009, unknown): “d1/n) 1/2 6 R 2 λn1/2 , and thus we requires that λn is large, which is a common assumption in regularized risk minimization [28, 30].”