Aggregation for Gaussian regression

By Florentina Bunea, Alexandre B. Tsybakov, Marten H. Wegkamp. 2007

366 citations recorded by the source corpus.

This paper studies statistical aggregation procedures in the regression setting. A motivating factor is the existence of many different methods of estimation, leading to possibly competing estimators. We consider here three different types of aggregation: model selection (MS) aggregation, convex (C) aggregation and linear (L) aggregation. The objective of (MS) is to select the optimal single estimator from the list; that of (C) is to select the optimal convex combination of the given estimators; and that of (L) is to select the optimal linear combination of the given estimators. We are interested in evaluating the rates of convergence of the excess risks of the estimators obtained by these procedures. Our approach is motivated by recently published minimax results [Nemirovski, A. (2000). Topics in non-parametric statistics. Lectures on Probability Theory and Statistics (Saint-Flour, 1998). Lecture Notes in Math. 1738 85-277. Springer, Berlin; Tsybakov, A. B. (2003). Optimal rates of aggregation. Learning Theory and Kernel Machines. Lecture Notes in Artificial Intelligence 2777 303-313. Springer, Heidelberg]. There exist competing aggregation procedures achieving optimal convergence rates for each of the (MS), (C) and (L) cases separately. Since these procedures are not directly comparable with each other, we suggest an alternative solution. We prove that all three optimal rates, as well as those for the newly introduced (S) aggregation (subset selection), are nearly achieved via a single "universal" aggregation procedure. The procedure consists of mixing the initial estimators with weights obtained by penalized least squares. Two different penalties are considered: one of them is of the BIC type, the second one is a data-dependent l 1 -type penalty.

Subfields: Artificial Intelligence, Management Science and Operations Research, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Best Subset Selection via a Modern Optimization Lens (2015, background): “In terms of the expected (worst case) predictive risk, it is well known [13, 47, 59] that the following upper bound holds:”
Square-Root Lasso: Pivotal Recovery of Sparse Signals via Conic Programming (2010, neutral): “imensional sparse models. Ann. Statist., 39(1):82–130, 2011. [4] P. J. Bickel, Y. Ritov, and A. B. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. Ann. Statist., 37(4):1705–1732, 2009. [5] F. Bunea, A. B. Tsybakov, and M. H. Wegkamp. Aggregation for Gaussian regression. Ann. Statist.,”
On the conditions used to prove oracle results for the Lasso (2009, background): “S-coherence conditions imply adaptive (S, s)-restricted regression conditions The papers [3, 5, 4] establish oracle results under a condition which we refer to as the restricted diagonal condition.”
Sup-norm convergence rate and sign concentration property of Lasso and Dantzig estimators (2008, background): “Assumption 2 is stated in a slightly weaker form in [1]-[4].”
Pivotal estimation via square-root Lasso in nonparametric regression (2011, background): “We refer to [6, 8, 9, 7, 12, 16, 17, 24, 30] for a more detailed review of the existing literature which has been focusing on the homoskedastic case.”
The Smooth-Lasso and other ℓ1+ℓ2-penalized methods (2011, background): “The results provided in Theorem 1 suggest essentially three regimes: • when φ 0 is a constant: in this case, the rate |A ∗ | log( p ) /n is optimal (up to a logarithmic factor; cf. [6, Theorem 5.1]).”
High-dimensional regression with unknown variance (2011, methodology): “This is, for instance, the case for the lasso [24, 72], Dantzig Selector [22], Elastic Net [85], MC+ [81], aggregation techniques [21, 26], etc.”
LASSO-TYPE RECOVERY OF SPARSE REPRESENTATIONS FOR HIGH-DIMENSIONAL DATA (2008, background): “Some studies Bunea, Tsybakov and Wegkamp, for example, [2], Greenshtein and Ritov, for example, [13], van de Geer, for example, [34] have focused mainly on the behavior of prediction loss.”
From Local Kernel to Nonlocal Multiple-Model Image Denoising (2009, background): “A ﬂow of publications on aggregation can be seen in mathematical statistics (e.g., Birge 2006; Bunea et al. 2007; Goldenshluger 2009).”
Taking Advantage of Sparsity in Multi-Task Learning (2009, background): “In fact, it has been recently shown by different authors, under different conditions on the design matrix, that the Lasso satisfies sparsity oracle inequalities, see [3, 5, 6, 26] and references therein.”
Oracle Inequalities and Optimal Inference under Group Sparsity (2010, background): “In particular, it has been shown by different authors, under different conditions on the design matrix, that the Lasso satisfies sparsity oracle inequalities; see [4, 6, 7, 18, 21, 36, 41] and references therein.”
NL-SAR: A Unified Nonlocal Framework for Resolution-Preserving (Pol)(In)SAR Denoising (2015, background): “Building a new estimate out of several is a well-studied problem in statistics referred to as “aggregation” [57].”
Assumptionless consistency of the Lasso (2013, background): “[5, 6], Candès and Tao [7, 8], Zhang and Huang [32], Lounici [21], Bickel et.”
Sparse Modeling for Image and Vision Processing (2014, background): “However, this highdimensional phenomenon currently6 comes at a price: (1) either an exhaustive search over the subsets of size k needs to be performed [Massart, 2003, Bunea et al., 2007, Raskutti et al., 2011] or (2) some assumptions have to be made regarding the dictionary D, which we now describe.”