Boosting for high-dimensional linear models

By Peter Buhlmann. 2006

450 citations recorded by the source corpus.

We prove that boosting with the squared error loss, L 2 Boosting, is consistent for very high-dimensional linear models, where the number of predictor variables is allowed to grow essentially as fast as O(exp(sample size)), assuming that the true underlying regression function is sparse in terms of the 11 -norm of the regression coefficients. In the language of signal processing, this means consistency for de-noising using a strongly overcomplete dictionary if the underlying signal is sparse in terms of the l 1 -norm. We also propose here an AIC-based method for tuning, namely for choosing the number of boosting iterations. This makes L 2 Boosting computationally attractive since it is not required to run the algorithm multiple times for cross-validation as commonly used so far. We demonstrate L 2 Boosting for simulated data, in particular where the predictor dimension is large in comparison to sample size, and for a difficult tumor-classification problem with gene expression microarray data.

Subfields: Computational Mechanics, Control and Systems Engineering, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3c9ab9694f688c2db62919526457ea56f1ea44f0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Adaptive Lasso for sparse high-dimensional regression models (2008, background): “Covariates with nonzero coefficients (1-5, 11-15, 21-25) are correlated with the rest.”
Flash-flood hazard assessment using ensembles and Bayesian-based machine learning models: Application of the simulated annealing feature selection method. (2019, methodology): “13 For fitting generalized linear model (GLM) (Lee and Nelder, 2006), GLMBoost generally uses a 264 functional gradient descent method for optimizing the overall loss functions through implementing 265 component-wise least squares while the variable selection can be carried out simultaneously 266 (B”
Sparse Models and Methods for Optimal Instruments With an Application to Eminent Domain (2012, methodology): “Bühlmann, P. (2006): “Boosting for high-dimensional linear models,” Ann. Statist., 34(2), 559–583. Bühlmann, P., and S. van de Geer (2011): Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer. Bunea, F., A. Tsybakov, and M. H. Wegkamp (2007a): “Sparsity oracle inequaliti”
A Panel Data Approach for Program Evaluation — Measuring the Benefits of Political and Economic Integration of Hong Kong with Mainland China (2012, methodology): “An alternative is to use a boosting method (Buhlmann, 2006) or LASSO (Tibshirani, 1996) or to use some targeted predictors based on soft and hard-thresholding as suggested by a referee.”
Selecting Instrumental Variables in a Data Rich Environment (2009, background): “Zhang and Yu (2005) and Buhlmann (2006) showed that boosting will consistently estimate the true conditional mean of x2.”
Machine Learning Advances for Time Series Forecasting (2020, background): “The Bagging algorithm is defined as follows.”
Stock market volatility: Identifying major drivers and the nature of their impact (2015, methodology): “For the linear model, consistency of L 2 -boosting in prediction norm was shown in Bühlmann (2006).”
Gradient boosting machines, a tutorial (2013, background): “More details on the properties of GLM and GAM boosting models can be found in Buhlmann (2006) and Schmid and Hothorn (2007).”
Statistical significance in high-dimensional linear models (2012, methodology): “The adaptive Lasso (Zou, 2006), analyzed in the high-dimensional setting by Huang, Ma and Zhang (2008) and van de Geer, B¨uhlmann and Zhou (2011), can be interpreted as an approximation of some concave penalization approach (Zou and Li, 2008).”
An Introduction to Recursive Partitioning: Rationale, Application and Characteristics of Classification and Regression Trees, Bagging and Random Forests (2009, unknown): “(2006), that high-dimensional data should be approached by several different methods because each single method has its strengths and weaknesses: Boosting, for example, can be used for variable selection in linear and other additive models (see Bühlmann, 2006; Bühlmann & Hothorn, 2007, for an implem”
On asymptotically optimal confidence regions and tests for high-dimensional models (2013, background): “The vast majority of work has been pursued for point estimation such as consistency for prediction [7, 21], oracle inequalities and estimation of a highdimensional parameter [6, 11, 12, 24, 33, 34, 47, 51] or variable selection [17, 30, 49, 53].”
Forward stagewise regression and the monotone lasso (2007, methodology): “On a more theoretical note, Buhlmann (2006) proves consistency of forward stagewise for high-dimensional linear models.”