Feature selection via regularized trees

By Houtao Deng, George C. Runger. 2012

227 citations recorded by the source corpus.

We propose a tree regularization framework, which enables many tree models to perform feature selection efficiently. The key idea of the regularization framework is to penalize selecting a new feature for splitting when its gain (e.g. information gain) is similar to the features used in previous splits. The regularization framework is applied on random forest and boosted trees here, and can be easily applied to other tree models. Experimental studies show that the regularized trees can select high-quality feature subsets with regard to both strong and weak classifiers. Because tree models can naturally deal with categorical and numerical variables, missing values, different scales between variables, interactions and nonlinearities etc., the tree regularization framework provides an effective and efficient feature selection solution for many practical problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Iterative random forests to discover predictive and stable high-order interactions (2017, background): “We note that other forms of regularization such as (Deng and Runger, 2012) may also lead to improved interaction recovery, though we do not explore them in this paper.”
Mathematical optimization in classification and regression trees (2021, methodology): “The regularization framework in Deng and Runger (2012) adds to the criterion optimized in each branch node a penalty term for predictor variables that have not appeared yet in the tree, so that the process is reluctant to use too many predictor variables, yielding a sparse tree model.”
Step away from stepwise (2018, methodology): “For example, good results have been reported with recursive feature elimination with cross-validation [38, 39] and regularized tree ensembles [40], which are two efficient ways of identifying a parsimonious set of predictors.”
Graph-based approach for airborne light detection and ranging segmentation (2014, methodology): “RRF t learns a regularized random forest (Deng and Runger, 2012) using caret as interface to the function RRF in the RRF package, with mtry=2 and tuning parameters coefReg={0.”
A random forest guided tour (2015, background): “Deng and Runger (2012) propose a Regularized Random Forest (RRF), which penalizes selecting a new feature for splitting when its gain is similar to the features used in previous splits. Deng and Runger (2013) suggest a Guided RRF (GRRF), in which the importance scores from an ordinary random forest ”
Do we need hundreds of classifiers to solve real world classification problems? (2014, methodology): “RRF t learns a regularized random forest (Deng and Runger, 2012) using caret as interface to the function RRF in the RRF package, with mtry=2 and tuning parameters coefReg={0.”
Big Data Analytics for Dynamic Energy Management in Smart Grids (2015, methodology): “The techniques that can be used to build an optimal subset for the load predicting problem in SG are greedy hill climbing [72], minimum-redundancy–maximum-relevance [73], regularized trees [74], random multinomial logit [75], etc.”
Sparsity in Optimal Randomized Classification Trees (2020, background): “Nonetheless, some attempts have been made, see [11, 12].”
Gene selection with guided regularized random forest (2012, methodology): “Therefore, RRF(1) not only improves interpretability by reducing the number of features, but also can improve the accuracy performance of classiﬁcation, even for RF, considered as a strong classiﬁer capable of handling irrelevant and relevant variables [4].”
Robustness of Random Forest-based gene selection methods (2013, methodology): “In order to make comparison meaningful, all calculations were performed on a homogeneous cluster of AMD Opteron 835X x86_64 Linux machines, using R 2.15.0 [26], randomForest 4.6-6, rFerns 0.3.1, RRF 1.2 and RF-ACE 1.1.0.”
A Brief Review of Random Forests for Water Scientists and Practitioners and Their Recent History in Water Resources (2019, background): “Regularized random forests [123,124] Improvements on variable selection within trees.”