Bias in random forest variable importance measures: Illustrations, sources and a solution

By Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn. 2007

1,433 citations recorded by the source corpus.

BackgroundVariable importance measures for random forests have been receiving increased attention as a means of variable selection in many classification tasks in bioinformatics and related scientific fields, for instance to select a subset of genetic markers relevant for the prediction of a certain disease. We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories. This is particularly important in genomics and computational biology, where predictors often include variables of different types, for example when predictors include both sequence data and continuous variables such as folding energy, or when amino acid sequence data show different numbers of categories.ResultsSimulation studies are presented illustrating that, when random forest variable importance measures are used with data of varying types, the results are misleading because suboptimal predictor variables may be artificially preferred in variable selection. The two mechanisms underlying this deficiency are biased variable selection in the individual classification trees used to build the random forest on one hand, and effects induced by bootstrap sampling with replacement on the other hand.ConclusionWe propose to employ an alternative implementation of random forests, that provides unbiased variable selection in the individual classification trees. When this method is applied using subsampling without replacement, the resulting variable importance measures can be used reliably for variable selection even in situations where the potential predictor variables vary in their scale of measurement or their number of categories. The usage of both random forest algorithms and their variable importance measures in the R system for statistical computing is illustrated and documented thoroughly in an application re-analyzing data from a study on RNA editing. Therefore the suggested method can be applied straightforwardly by scientists in bioinformatics research.

Subfields: Artificial Intelligence, Molecular Biology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Statistics and Probability

Citations in the corpus

A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data (2009, methodology): “Measuring feature relevance using the Gini importance is subject to selection bias on factorial data [30].”
Boruta – A System for Feature Selection (2010, background): “It has been shown by simulations [16, 17, 18] and observation of experimental data [8] that this assumption may be false for some data sets.”
Permutation importance: a corrected feature importance measure (2010, methodology): “A recent study showed that, if predictors are categorical, both measures are biased in favor of variables taking more categories (Strobl et al., 2007).”
A standard protocol for reporting species distribution models (2020, methodology): “We encourage authors to report whether and how model coefficients were extracted from the models and analysed, and how variable importance was determined (e.g. through permutation, Strobl et al. 2007).”
An Introduction to Recursive Partitioning: Rationale, Application and Characteristics of Classification and Regression Trees, Bagging and Random Forests (2009, positive): “This bias is carried forward to ensembles of trees: Especially the variable importance can be biased when a data set contains predictor variables of different types (Strobl et al. 2007).”
Prediction of In-hospital Mortality in Emergency Department Patients With Sepsis: A Local Big Data-Driven, Machine Learning Approach. (2016, methodology): “48 Following the methods of Strobl et al., 49 individual conditional inference trees were constructed using bootstrap samples of the full data set.”
Random Forests for Genomic Data Analysis (2012, background): “their scale of measurement or their number of categories [11].”
VSURF: An R Package for Variable Selection Using Random Forests (2015, positive): “Strobl et al. (2007) showed that VI scores are biased towards correlated variables, and Strobl et al. (2008) proposed an alternative permutation scheme as a solution, which, however, increases the computation cost.”
Relevant Feature Set Estimation with a Knock-out Strategy and Random Forests (2015, neutral): “Various importance measures can be used for this purpose, such as Gini importance and permutation importance (Strobl et al., 2007), and each of thesemeasures can be used to rank features based on their importance.”
Computational Statistics and Data Analysis a Similarity Measure to Assess the Stability of Classification Trees (neutral): “A description and comparison of these measures can be found in Archer and Kimes (2008) and Strobl et al. (2007).”
Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics (2012, methodology): “The alternative class of decision trees developed by Hothorn et al. [36] and Strobl et al. [68] addresses this issue through the principle of conditional hypothesis testing.”
Support Vector Machine Versus Random Forest for Remote Sensing Image Classification: A Meta-Analysis and Systematic Review (2020, positive): “They have been widely adopted and applied as a standard clas-siﬁer to a variety of prediction and classiﬁcation tasks, such as those in bioinformatics [131], computer vision [132], and RS land cover classiﬁcation [133].”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, methodology): “2 The random forest paper has been cited over 60,000 times (Google Scholar; September 2020) and there are many papers improving the importance measure ([110,111,44,56]) which are also cited frequently.”
Variable Importance Assessment in Regression: Linear Regression versus Random Forest (2009, methodology): “A further instance proposed by Strobl et al. (2007), RF-CI, will also be introduced below.”