Gene selection with guided regularized random forest

By Houtao Deng, George C. Runger. 2012

304 citations recorded by the source corpus.

The regularized random forest (RRF) was recently proposed for feature selection by building only one ensemble. In RRF the features are evaluated on a part of the training data at each tree node. We derive an upper bound for the number of distinct Gini information gain values in a node, and show that many features can share the same information gain at a node with a small number of instances and a large number of features. Therefore, in a node with a small number of instances, RRF is likely to select a feature not strongly relevant. Here an enhanced RRF, referred to as the guided RRF (GRRF), is proposed. In GRRF, the importance scores from an ordinary random forest (RF) are used to guide the feature selection process in RRF. Experiments on 10 gene data sets show that the accuracy performance of GRRF is, in general, more robust than RRF when their parameters change. GRRF is computationally efficient, can select compact feature subsets, and has competitive accuracy performance, compared to RRF, varSelRF and LASSO logistic regression (with evaluations from an RF classifier). Also, RF applied to the features selected by RRF with the minimal regularization outperforms RF applied to all the features for most of the data sets considered here. Therefore, if accuracy is considered more important than the size of the feature subset, RRF with the minimal regularization may be considered. We use the accuracy performance of RF, a strong classifier, to evaluate feature selection methods, and illustrate that weak classifiers are less capable of capturing the information contained in a feature subset. Both RRF and GRRF were implemented in the ''RRF'' R package available at CRAN, the official R package archive.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Molecular Biology

Citations in the corpus

Interpreting tree ensembles with inTrees (2014, methodology): “699 X[,12] %in% c(’N’) & X[,5] %in% c(’Y’) win 2 0.”
Robustness of Random Forest-based gene selection methods (2013, methodology): “Regularized Random Forest (RRF) is a modification of Random Forest which incorporates regularisation into tree growing algorithm [9, 14]; namely, it establishes a penalty to use an attribute not previously used in a current tree construction.”
Guided Random Forest in the RRF Package (2013, methodology): “I obtained the average error rates and average number of features for each method using the same procedure as Deng and Runger (2013), i.e., calculated from 100 replicates of training/testing splits with a ratio of 2:1.”
A Comparison of Random Forest Variable Selection Methods for Classification Prediction Modeling (2019, methodology): “A stepwise selection procedure is implemented in VSURF (Genuer, et al., 2015), whereas RRF (Deng & Runger, 2013) uses a regularized random forest procedure and a forward selection approach.”
Sparsity in Optimal Randomized Classification Trees (2020, methodology): “This approach is refined in [12], by also including the importance scores of the predictor variables, obtained in a preprocessing step running a preliminary RF.”
CBC: An associative classifier with a small number of rules (2014, background): “[7] discussed the node sparsity issue in decision tree algorithms.”
A random forest guided tour (2015, methodology): “Deng and Runger (2013) suggest aGuided RRF (GRRF), inwhich the importance scores from an ordinary random forest are used to guide the feature selection process in RRF. Lastly, a Garrote-style convex penalty, proposed by Meinshausen (2009), selects functional groups of nodes in trees, yielding to…”
A Brief Review of Random Forests for Water Scientists and Practitioners and Their Recent History in Water Resources (2019, background): “Regularized random forests [123,124] Improvements on variable selection within trees.”