The behaviour of random forest permutation-based variable importance measures under predictor correlation

By Kristin K. Nicodemus, James D. Malley, Carolin Strobl, Andreas Ziegler. BMC Bioinformatics, 2010

351 citations recorded by the source corpus.

BackgroundRandom forests (RF) have been increasingly used in applications such as genome-wide association and microarray studies where predictor correlation is frequently observed. Recent works on permutation-based variable importance measures (VIMs) used in RF have come to apparently contradictory conclusions. We present an extended simulation study to synthesize results.ResultsIn the case when both predictor correlation was present and predictors were associated with the outcome (HA), the unconditional RF VIM attributed a higher share of importance to correlated predictors, while under the null hypothesis that no predictors are associated with the outcome (H0) the unconditional RF VIM was unbiased. Conditional VIMs showed a decrease in VIM values for correlated predictors versus the unconditional VIMs under HA and was unbiased under H0. Scaled VIMs were clearly biased under HA and H0.ConclusionsUnconditional unscaled VIMs are a computationally tractable choice for large datasets and are unbiased under the null hypothesis. Whether the observed increased VIMs for correlated predictors may be considered a "bias" - because they do not directly reflect the coefficients in the generating model - or if it is a beneficial attribute of these VIMs is dependent on the application. For example, in genetic association studies, where correlation between markers may help to localize the functionally relevant variant, the increased importance of correlated predictors may be an advantage. On the other hand, we show examples where this increased importance may result in spurious signals.

Subfields: Genetics, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/180e6da4810a53defc935db2ea63f6191c5d459b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

Citations in the corpus

ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R (2015, methodology): “The permutation importance can be obtained unnormalized, as recommended by Nicodemus et al. (2010), or scaled by their standard errors (Breiman 2001).”
Data mining in the Life Sciences with Random Forest: a walk in the park or lost in the jungle? (2012, background): “Variable importance measures are currently subject of debate and rankings produced using permutation importance may be preferred over Gini importance rankings when variables: (i) are correlated [105, 108–110], (ii) vary in their scale of measurement (e.”
A new variable importance measure for random forests with missing data (2012, background): “The discussions in Nicodemus et al. (2010) and Altmann et al. (2010) show that both kinds of measures, conditional and unconditional, can be of speciﬁc value depending on the research question.”
Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance (2020, background): “…(oob) (Breiman, 2001) importance scores are one such popular measure, though many issues such as a preference for correlated features and those with many categories have been noted in the years following their introduction (Strobl et al., 2007; Nicodemus et al., 2010; Tolo¸si and Lengauer, 2011).”
Identifying appropriate spatial scales of predictors in species distribution models with the random forest algorithm (2013, methodology): “However, the permutation importance for noncausal predictors, which were correlated with causal predictors was inflated (Strobl et al. 2008;Nicodemus et al. 2010).”
Grouped variable importance with random forests and application to multiple functional data analysis (2014, background): “Indeed, it has been noticed by Nicodemus et al. (2010) that the individual importances of predictors that are not associated with the outcome tend to be overestimated by the random forests, when there is correlation between predictors.”
Random Forests for Genomic Data Analysis (2012, methodology): “Nicodemus et al. [43] performed simulation studies to compare conditional permutation VIMP with standard permutation VIMP.”
Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics (2012, methodology): “[58] explore the behavior of the permutation VIM in the presence of correlated predictors in an extensive simulation study based on data generated from the logistic regression model.”
On Oblique Random Forests (2011, background): “It has been observed that correlation between features may affect both Gini [23] and permutation importance [28].”
Please Stop Permuting Features: An Explanation and Alternatives (2019, background): “Nicodemus et al. (2010) investigates these claims of feature preference in a large scale simulation study and again find that the oob measures overestimate the importance of correlated predictors.”
Object-oriented mapping of landslides using Random Forests (2011, background): “The properties of those measures have been intensively studied in recent years, and the so-called permutation importance is considered a computationally tractable choice for the screening of large datasets (Nicodemus et al., 2010).”
Correlation and variable importance in random forests (2013, background): “[30], Nicodemus [28], Auret and Aldrich [3] and Toloşi and Lengauer [37].”