Sparse additive models

By Pradeep Ravikumar, John D. Lafferty, Han Liu, Larry A. Wasserman. 2007

617 citations recorded by the source corpus.

Summary. We present a new class of methods for high dimensional non‐parametric regression and classification called sparse additive models. Our methods combine ideas from sparse linear modelling and additive non‐parametric regression. We derive an algorithm for fitting the models that is practical and effective even when the number of covariates is larger than the sample size. Sparse additive models are essentially a functional version of the grouped lasso of Yuan and Lin. They are also closely related to the COSSO model of Lin and Zhang but decouple smoothing and sparsity, enabling the use of arbitrary non‐parametric smoothers. We give an analysis of the theoretical properties of sparse additive models and present empirical results on synthetic and real data, showing that they can be effective in fitting sparse non‐parametric models in high dimensional data.

Subfields: Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…functions are estimated by splines, many works apply convex regularizers (e.g., `1) to control both smoothness and sparsity (Lin et al., 2006; Ravikumar et al., 2009; Meier et al., 2009; Yin and Yu, 2017; Petersen et al., 2016; Lou et al., 2016; Sadhanala et al., 2019; Haris et al., 2019).”
Generalized Sparse Additive Models (2019, methodology): “However, previous proposals consists of gaps around eﬃcient computation [Koltchinskii and Yuan, 2010, Raskutti et al., 2012, Yuan and Zhou, 2015] and/or optimal statistical convergence properties [Ravikumar et al., 2009, Lou et al., 2016, Petersen et al., 2016, Sadhanala and Tibshirani, 2018].”
Additive models with trend filtering (2017, background): “2005), which is known as 1d total variation denoising in signal processing (Rudin et al. 1992). Over the kth order total variation function class defined in (4), Tibshirani (2014), Wang et al. (2014) prove that kth order trend filtering achieves the minimax optimal n−(2k+2)/(2k+3) error rate, just l”
Fused Lasso Additive Model (2014, methodology): “These are the exact functions used for the simulations in the original SpAM paper (Ravikumar et al. 2009).”
Sparse Partially Linear Additive Models (2014, methodology): “5.2 SIMULATION: EFFECT OF |N | AND |L| ON SPLAM, SPAM, AND THE LASSO In this section, we perform a large-scale simulation to gain deeper insights into the lasso (Tibshirani 1996), SPLAM, and SpAM (Ravikumar et al. 2009).”
CAM: Causal Additive Models, high-dimensional order search and penalized regression (2013, methodology): “For example with the Group Lasso, assuming some condition avoiding near collinearity of functions, i.e., a compatibility condition for the Group Lasso (B¨uhlmann and van de Geer, 2011, Ch.5.6,Th.8.2) and that the (cid:96) 2 -norms of the non-zero functions are suﬃciently large, we obtain the screeni”
Optimization with Sparsity-Inducing Penalties (2011, methodology): “A block-coordinate algorithm can be applied by considering each RKHS Hi as one “block”; this type of algorithm was considered by [99].”
Robust Lasso With Missing and Grossly Corrupted Observations (2011, background): “, [45] and [46]) under grossly corrupted observations.”
The Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs (2009, methodology): “Under appropriate sparsity conditions, the resulting estimator Ω has been shown to have good theoretical properties (Rothman et al., 2008; Ravikumar et al., 2009).”
ABC–CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations (2018, methodology): “jx0 j )) 2dxdx0 j d; and its relevance to the regression i(x) in Equation1as r i;j := ZZ 0 i(x) 0 i(x j ) 2 dxdx j : 8FlexCode with the coecients from Equation1estimated via Sparse Additive Models (Ravikumar et al., 2009) 9FlexCode with the coecients from Equation1estimated via Random Forests 14 Und”
Semiparametric regression models with additive nonparametric components and high dimensional parametric components (2012, methodology): “The proof of Proposition 2.2 is a straightforward extension to Theorem 1 in Ravikumar et al. (2009) and thus omitted here.”
High Dimensional Bayesian Optimisation and Bandits via Additive Models (2015, background): “GP-UCB on D th order kernel”
Wavelet regression and additive models for irregularly spaced data (2019, neutral): “For a real world data analysis, we consider the Boston housing data analyzed by Ravikumar et al. [2009]. The goal is to predict the median value of homes based on 10 predictors.”
Canonical Correlation Analysis based on Hilbert-Schmidt Independence Criterion and Centered Kernel Target Alignment (2013, background): “(Balakrishnan et al., 2012) provides an attempt to address these issues by utilizing non-linear transformations that are modelled by sparse generalized additive models (Ravikumar et al., 2009).”
Intelligible models for classification and regression (2012, methodology): “Generalized additive models were introduced by the statistics community [14, 15, 22] and have been extended to include LASSO feature selection [18] and to incorporate interaction terms [16].”
Sparse kernel canonical correlation analysis for discovery of nonlinear interactions in high-dimensional data (2017, methodology): “Other researchers have employed the sparse additive model [13] to extend KCCA to high-dimensional problems, and have defined two equivalent formulations, such as sparse additive functional CCA (SAFCCA) and sparse additive kernel CCA (SAKCCA) [12].”
Partially Linear Functional Additive Models for Multivariate Functional Data (2018, methodology): “This is a fundamental difference with classic high-dimensional additive models (Meier, van de Geer, and Bühlmann 2009; Ravikumar et al. 2009; Liu,Wang, and Liang 2011;Wang et al. 2014).”
Sparsistency and Rates of Convergence in Large Covariance Matrix Estimation (2007, background): “By sparsistency, we mean the property that all parameters that are zero are actually estimated as zero with probability tending to one, a weaker requirement than that of Ravikumar et al. (2007).”