High-Dimensional Feature Selection by Feature-Wise Kernelized Lasso

By Makoto Yamada, Wittawat Jitkrittum, Leonid Sigal, Eric P. Xing, Masashi Sugiyama. 2012

344 citations recorded by the source corpus.

The goal of supervised feature selection is to find a subset of input features that are responsible for predicting output values. The least absolute shrinkage and selection operator (Lasso) allows computationally efficient feature selection based on linear dependency between input features and output values. In this letter, we consider a feature-wise kernelized Lasso for capturing nonlinear input-output dependency. We first show that with particular choices of kernel functions, nonredundant features with strong statistical dependence on output values can be found in terms of kernel-based independence measures such as the Hilbert-Schmidt independence criterion. We then show that the globally optimal solution can be efficiently computed; this makes the approach scalable to high-dimensional problems. The effectiveness of the proposed method is demonstrated through feature selection experiments for classification and regression with thousands of features.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

Citations in the corpus

Causal Generative Neural Networks (2017, methodology): “Following [Tsamardinos et al. , 2006; Nandy et al. , 2015], we assume known skeleton for G , obtained via domain knowledge or a feature selection algorithm [Yamada et al. , 2014] under standard assumptions such as causal Markov, faithfulness, and sufﬁciency.”
Explainable and Interpretable Models in Computer Vision and Machine Learning (2017, neutral): “Taking inspiration from Tsamardinos et al. (2006); Nandy et al. (2015), we start from a graph skeleton recovered by other methods such as feature selection (Yamada et al., 2014).”
Deep Neural Networks for High Dimension, Low Sample Size Data (2017, methodology): “We compare the proposed DNP method with three representative feature selection algorithms, including l1-penalized logistic regression (LogR-l1), gradient boosted feature selection (GBFS) [Xu et al., 2014], and HSIC-Lasso [Yamada et al., 2014].”
Explainability in Graph Neural Networks: A Taxonomic Survey (2020, methodology): “Then a nonlinear surrogate model, Hilbert-Schmidt Independence Criterion (HSIC) Lasso [73], is employed to fit the local dataset.”
Sparse kernel canonical correlation analysis for discovery of nonlinear interactions in high-dimensional data (2017, methodology): “Feature-wise kernel was introduced to perform feature-wise nonlinear Lasso [20].”
New improvements in the use of dependence measures for sensitivity analysis and screening (2014, background): “[29] propose the HSIC Lasso which consists in the minimization of this objective function with D(Y) = HKH and D(Xk) = HKkH, ∀k ∈ {1, .”
GraphLIME: Local Interpretable Model Explanations for Graph Neural Networks (2020, methodology): “The explanation model class G used in this paper is Hilbert-Schmidt Independence Criterion Lasso (HSIC Lasso) [23, 24, 25], which is a kernel based nonlinear interpretable feature selection algorithm.”
Convolutional Neural Networks-Based MRI Image Analysis for the Alzheimer’s Disease Prediction From Mild Cognitive Impairment (2018, methodology): “LASSO is a supervised learning method that uses L 1 norm in sparse regression (Kukreja et al., 2006; Usai et al., 2009; Yamada et al., 2014) as follows: Where y ∈ R × N is the vector consisting of N labels of training samples, D ∈ R N × M is the feature matrix of N training samples consisting of M…”