Component selection and smoothing in multivariate nonparametric regression

By Yi Lin, Hao Helen Zhang. 2006

613 citations recorded by the source corpus.

We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The "COSSO" is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

Subfields: Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “, 1) to control both smoothness and sparsity [186, 245, 206, 324, 234, 194, 262, 121].”
Additive models with trend filtering (2017, background): “Lin and Zhang [2006], Ravikumar et al. [2009], Meier et al. [2009], Koltchinskii and Yuan [2010], Raskutti et al.”
Fused Lasso Additive Model (2014, neutral): “Flexible additive modeling in high dimensions has been an active area of research in recent years (Bühlmann and Yu 2003; Sardy and Tseng 2004; Lin and Zhang 2006; Avalos, Grandvalet, and Ambroise 2007; Bühlmann and Hothorn 2007; Huang, Horowitz, and Wei 2010; Zhang, Cheng, and Liu 2011; Wood,…”
Sparse Partially Linear Additive Models (2014, methodology): “An alternative approach to SpAM is the component selection and smoothing operator (COSSO) method, which uses unsquared reproducing kernel Hilbert space (RKHS) norm penalties (Lin and Zhang 2006).”
Breaking the Curse of Dimensionality with Convex Neural Networks (2014, methodology): “The set A may be either given (Gu, 2013) or learned from data (Lin and Zhang, 2006; Bach, 2008b).”
Sparse additive models (2007, methodology): “As discussed in the introduction, the COSSO model of Lin and Zhang (2006) replaces the lasso constraint on ∑ j |βj | with a RKHS constraint.”
High-dimensional additive modeling (2008, methodology): “This is similar to Example 2 in [19] but with fewer observations.”
Generalized Additive Model Selection (2015, methodology): “The methods most closely related to our proposal are COSSO (Lin et al., 2006), SpAM (Ravikumar et al., 2007), and the method of Meier et al. (2009).”
ANOVA kernels and RKHS of zero mean functions for model-based sensitivity analysis (2011, background): “One can note that the analytical value is at the most one standard deviation away from the estimated mean.”
A UNIFIED VARIABLE SELECTION APPROACH FOR VARYING COEFFICIENT MODELS (2012, methodology): “The COSSO was first studied by Lin and Zhang (2006) for additive models, and it was extended by Leng (2009) to varying coefficient models.”
Nonparametric sparsity and regularization (2012, methodology): “While for MKL and `1-features we are able to identify the set of selected variables, for COSSO and HKL extracting the sparsity patterns from the available code is not straightforward.”
Doubly penalized estimation in additive regression with high-dimensional data (2019, background): “…settings where p can be of greater order than the sample size n but the number of significant components is still smaller than n; see, for example, Lin and Zhang (2006), Meier, van de Geer and Bühlmann (2009), Ravikumar et al. (2009), Huang, Horowitz and Wei (2010), Koltchinskii and Yuan (2010),…”
Least angle and ℓ1 penalized regression: A review (2008, methodology): “Lin and Zhang (2006) let the groups of predictors correspond to sets of basis functions for smoothing splines, in which the penalty ‖βj‖Kj would give the square-root of the integrated squared second derivative of a spline function (a linear combination of the basis functions).”