Gradient boosted feature selection

By Zhixiang Eddie Xu, Gao Huang, Kilian Q. Weinberger, Alice X. Zheng. 2014

204 citations recorded by the source corpus.

A feature selection algorithm should ideally satisfy four conditions: reliably extract relevant features; be able to identify non-linear feature interactions; scale linearly with the number of features and dimensions; allow the incorporation of known sparsity structure. In this work we propose a novel feature selection algorithm, Gradient Boosted Feature Selection (GBFS), which satisfies all four of these requirements. The algorithm is flexible, scalable, and surprisingly straight-forward to implement as it is based on a modification of Gradient Boosted Trees. We evaluate GBFS on several real world data sets and show that it matches or outperforms other state of the art feature selection algorithms. Yet it scales to larger data set sizes and naturally allows for domain-specific side information.

Subfields: Computer Vision and Pattern Recognition, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Qiang Yang
2. Jiliang Tang
3. Jundong Li
4. Yu Zhang
5. Ying Wei
6. Fred Morstatter
7. Huan Liu
8. Suhang Wang

In Molecular Biology

1. Yu Zhang
2. Huan Liu
3. Qiang Yang
4. Ying Wei
5. Jiliang Tang
6. Jundong Li
7. Fred Morstatter
8. Suhang Wang

Citations in the corpus

• Automatic Model Monitoring for Data Streams (2019, background): “This allows us to: 1) obtain an alarm score that can be used to rank events in T (higher score⇒ closer to the top), and 2) directly obtain a measure of feature importance that deals well with correlated features [33].”
• Feature selection based on artificial bee colony and gradient boosting decision tree (2019, methodology): “The steps of GBDT [29,30] are presented as follows: Step 1: The initial constant value of the model β is given: Step 2: For the number of iterations m = 1 : M (M is the times of iteration), the gradient direction of residuals are calculated.”
• Global Model Interpretation Via Recursive Partitioning (2018, methodology): “Œe feature selection process could either be conducted before the model €Šing [11] or embedded into it [24, 27].”
• Deep Neural Networks for High Dimension, Low Sample Size Data (2017, methodology): “We compare the proposed DNP method with three representative feature selection algorithms, including l1-penalized logistic regression (LogR-l1), gradient boosted feature selection (GBFS) [Xu et al., 2014], and HSIC-Lasso [Yamada et al., 2014].”
• Comparison of Variable Selection Methods for Clinical Predictive Modeling (2018, methodology): “…regression-based methods (stepwise backward selection using p-value and AIC, Least Absolute Shrinkage and Selection Operator, and Elastic Net) and four tree-based methods (Variable Selection Using Random Forest, Regularized Random Forests, Boruta, and Gradient Boosted Feature Selection) [8, 19–23].”
• Feature Selection (2016, methodology): “This property makes it suitable for feature selection, as we can select features whose corresponding feature weights are large, which motivates a surge of 1-norm regularized feature selection methods (Zhu et al. 2004; Xu et al. 2014; Wei et al. 2016a; Wei and Yu 2016; Hara and Maehara 2017).”