Nonparametric sparsity and regularization

By Lorenzo Rosasco, Silvia Villa, Sofia Mosci, Matteo Santoro, Alessandro Verri. 2012

109 citations recorded by the source corpus.

In this work we are interested in the problems of supervised learning and variable selection when the input-output dependence is described by a nonlinear function depending on a few variables. Our goal is to consider a sparse nonparametric model, hence avoiding linear or additive models. The key idea is to measure the importance of each variable in the model by making use of partial derivatives. Based on this intuition we propose a new notion of nonparametric sparsity and a corresponding least squares regularization scheme. Using concepts and results from the theory of reproducing kernel Hilbert spaces and proximal methods, we show that the proposed learning algorithm corresponds to a minimization problem which can be provably solved by an iterative procedure. The consistency properties of the obtained estimator are studied both in terms of prediction and selection performance. An extensive empirical analysis shows that the proposed method performs favorably with respect to the state-of-the-art methods.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Optimal Rates for Random Fourier Features (2015, background): “As motivated in Section 1, the question of approximating the derivatives of the kernel through finite dimensioal random feature map is also important as it enables to speed up several interesting machine learning tasks that involve the derivatives of the kernel (e.g., [32, 21, 17, 18, 29, 23]).”
Variable selection in functional data classification: a maxima-hunting proposal (2013, methodology): “ the setting of regression models) to use the terms \sparse" or \sparsity" to describe situations in which variable selection is the rst natural aim; see e.g., Gertheiss and Tutz (2010) and Rosasco et al. (2013). It has been also argued in Kneip and Sarda (2011) that the standard sparsity ”
Learning Sparse Nonparametric DAGs (2019, methodology): “Also of relevance is the literature on nonparametric variable selection (Bertin et al., 2008; Lafferty et al., 2008; Miller et al., 2010; Rosasco et al., 2013; Gregorová et al., 2018) and approaches based on neural networks (Feng and Simon, 2017; Ye and Sun, 2018; Abid et al.”
Gradient Weights help Nonparametric Regressors (2012, background): “kernel or k-NN) is well known to be the sum of its variance and its bias [7].”
D’ya Like DAGs? A Survey on Structure Learning and Causal Discovery (2021, methodology): “(2013) [206]) such that [W( f )]k j := | |∂k fj | |2.”