Sparse Partially Linear Additive Models

By Yin Lou, Jacob Bien, R. Caruana, Johannes Gehrke. 2014

60 citations recorded by the source corpus.

The generalized partially linear additive model (GPLAM) is a flexible and interpretable approach to building predictive models. It combines features in an additive manner, allowing each to have either a linear or nonlinear effect on the response. However, the choice of which features to treat as linear or nonlinear is typically assumed known. Thus, to make a GPLAM a viable approach in situations in which little is known a priori about the features, one must overcome two primary model selection challenges: deciding which features to include in the model and determining which of these features to treat nonlinearly. We introduce the sparse partially linear additive model (SPLAM), which combines model fitting and both of these model selection challenges into a single convex optimization problem. SPLAM provides a bridge between the lasso and sparse additive models. Through a statistical oracle inequality and thorough simulation, we demonstrate that SPLAM can outperform other methods across a broad spectrum of statistical regimes, including the high-dimensional (p ≫ N) setting. We develop efficient algorithms that are applied to real datasets with half a million samples and over 45,000 features with excellent predictive performance. Supplementary materials for this article are available online.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Statistics and Probability

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “…functions are estimated by splines, many works apply convex regularizers (e.g., `1) to control both smoothness and sparsity (Lin et al., 2006; Ravikumar et al., 2009; Meier et al., 2009; Yin and Yu, 2017; Petersen et al., 2016; Lou et al., 2016; Sadhanala et al., 2019; Haris et al., 2019).”
Generalized Sparse Additive Models (2019, methodology): “However, previous proposals consists of gaps around eﬃcient computation [Koltchinskii and Yuan, 2010, Raskutti et al., 2012, Yuan and Zhou, 2015] and/or optimal statistical convergence properties [Ravikumar et al., 2009, Lou et al., 2016, Petersen et al., 2016, Sadhanala and Tibshirani, 2018].”
Additive models with trend filtering (2017, methodology): “For part (b), we use a special spline quasi-interpolant from Proposition 7 in Mammen & van de Geer (1997) (who in turn use results from de Boor (1978)); for part (c), we develop a new falling factorial approximant that may be of independent interest.”
Fused Lasso Additive Model (2014, methodology): “Recently, Lou et al. [2014] proposed the sparse partially linear additive model, which models a subset of the included features linearly and the remaining included features non-linearly using basis functions.”
Generalized Additive Model Selection (2015, methodology): “GAMSEL 23 experiments indicate that GAMSEL can have notably better selection performance when some of the underlying effects are in truth linear. Moreover, GAMSEL was observed to perform competitively even when the ground truth is that all effects are non-linear. We also showed that our method perfo”
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations (2020, background): “GAMSEL [10] and SPLAM [38] have formulated new optimization objectives to induce sparsity and partial linearity to find one GAM solution that has the best accuracy.”