High-dimensional additive modeling

By Lukas Meier, Sara A. van de Geer, Peter Buhlmann. 2008

502 citations recorded by the source corpus.

We propose a new sparsity-smoothness penalty for high-dimensional generalized additive models. The combination of sparsity and smoothness is crucial for mathematical theory as well as performance for finite-sample data. We present a computationally efficient algorithm, with provable numerical convergence properties, for optimizing the penalized likelihood. Furthermore, we provide oracle results which yield asymptotic optimality of our estimator for high dimensional but sparse additive models. Finally, an adaptive version of our sparsity-smoothness penalized approach yields large additional performance gains.

Subfields: Computational Mechanics, Numerical Analysis, Statistics and Probability, Probabilistic & Bayesian Methods, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Numerical Analysis

In Statistics and Probability

In Probabilistic & Bayesian Methods

In Statistical Learning Theory

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…functions are estimated by splines, many works apply convex regularizers (e.g., `1) to control both smoothness and sparsity (Lin et al., 2006; Ravikumar et al., 2009; Meier et al., 2009; Yin and Yu, 2017; Petersen et al., 2016; Lou et al., 2016; Sadhanala et al., 2019; Haris et al., 2019).”
Generalized Sparse Additive Models (2019, methodology): “Meier et al. [2009] give two proposals: the ﬁrst solves the optimization problem argmin f 1 ,...,f p and is not a GSAM ; they note that this proposal gives a suboptimal rate of convergence.”
Additive models with trend filtering (2017, background): “We do not pursue such a relaxation, but we refer the reader to Meier et al. (2009), van de Geer (2014) for similar strategies. Remark 9 (The role of sup norm boundedness). The sup norm bound ‖Πk,n(f0)‖∞ ≤ b0 in Assumption A2, as well as the sup norm constraint ‖Πk,n( ∑d j=1 fj)‖∞ ≤ b0 in the problem”
Fused Lasso Additive Model (2014, background): “Other variations of this sparsity-smoothness penalty have also been proposed [B¨uhlmann and Van De Geer, 2011, Meier et al., 2009].”
CAM: Causal Additive Models, high-dimensional order search and penalized regression (2013, methodology): “Instead of the boosting method, we could alternatively use additive model fitting with a sparsity- or sparsity-smoothness penalty [16, 28].”
Sparse additive models (2007, background): “References include Wainwright (2006), Meinshausen and Bühlmann (2006), Zou (2005), Fan and Li (2001), and Zhao and Yu (2007).”
Robust Lasso With Missing and Grossly Corrupted Observations (2011, background): “, [45] and [46]) under grossly corrupted observations.”
Group Sparse Additive Models (2012, background): “Finally, it is important to emphasize that our work is substantially diﬀerent from another line of research that also mixes group lasso and additive models (Bach, 2008; Meier et al., 2009; Huang et al., 2010).”
Doubly penalized estimation in additive regression with high-dimensional data (2019, result): “Fourth, the rate of convergence (6) under exact sparsity (q = 0) is known to be in general faster than in Meier et al. (2009). Compared with previous results giving similar rates of convergence as (6) with q = 0 for Hilbert spaces, our results are stronger in requiring much weaker technical conditio”
Generalized Additive Model Selection (2015, methodology): “The methods most closely related to our proposal are COSSO (Lin et al., 2006), SpAM (Ravikumar et al., 2007), and the method of Meier et al. (2009).”
Sparse Additive Ordinary Differential Equations for Dynamic Gene Regulatory Network Modeling (2014, methodology): “Thus, this is not a standard sparse additive regression model studied in the literature (Meier, van de Geer, and Buhlmann 2009; Ravikumar et al. 2009; Huang, Horowitz, and Wei 2010; Can-toni, Flemming, and Ronchetti 2011).”
Functional additive regression (2015, unknown): “04 06 4v 1 [ m at h. ST ] 1 4 O ct 2 01 5 The Annals of Statistics 2015, Vol. 43, No. 5, 2296–2325 DOI: 10.1214/15-AOS1346 c© Institute of Mathematical Statistics, 2015”
Weakly decomposable regularization penalties and structured sparsity (2012, methodology): “An example, concerning the high-dimensional additive model, is in the work of Meier et al. (2009).”
Finding and Leveraging Structure in Learning Problems (2012, background): “Recently, several authors have shown how sparse additive models for regression can be efficiently estimated even when p > n [115, 139, 156, 157].”
Nonparametric Independence Screening in Sparse Ultra-High Dimensional Additive Models (2009, methodology): “In fact, any variable selection method for additive models will work such as the SpAM in Ravikumar et al. (2009) and also the adaptive group LASSO for additive models in Huang et al. (2010).”