Variable importance in binary regression trees and forests

By Hemant Ishwaran. 2007

411 citations recorded by the source corpus.

We characterize and study variable importance (VIMP) and pairwise variable associations in binary regression trees. A key component involves the node mean squared error for a quantity we refer to as a maximal subtree. The theory naturally extends from single trees to ensembles of trees and applies to methods like random forests. This is useful because while importance values from random forests are used to screen variables, for example they are used to filter high throughput genomic data in Bioinformatics, very little theory exists about their properties.

Subfields: Genetics, Molecular Biology, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

In Statistics and Probability

Citations in the corpus

Consistent Individualized Feature Attribution for Tree Ensembles (2018, methodology): “Different choices about the method of feature value permutation lead to variations of this basic approach [1, 10, 14, 23, 26].”
Variable Importance Using Decision Trees (2017, background): “Additionally, due to the algorithmic complexity of the procedure, it is not immediately amenable to theoretical analysis, though the asymptotic properties of a simplified variant of the procedure have been studied in [6].”
Explainable AI for Trees: From Local Explanations to Global Understanding (2019, background): “While local explanation methods for trees have not been extensively studied, interpreting tree-based models by assigning a global importance value to each input feature is a well studied problem and many methods have been proposed [20, 19, 61, 69, 70, 27, 2, 40, 31].”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, background): “ample, [123,66,52]). Both regression models and 2 The random forest paper has been cited over 60,000 times (Google Scholar; September 2020) and there are many papers improving the importance measure ([110,111,44,56]) which are also cited frequently. IML - History, Methods, Challenges 3 Fig.1. Left: ”
Understanding variable importances in forests of randomized trees (2013, methodology): “Little is known however regarding the variable importances computed by Random Forests like algorithms, and – as far as we know – the work of Ishwaran (2007) is indeed the only theoretical study of tree-based variable importance measures.”
A random forest based approach for predicting spreads in the primary catastrophe bond market (2020, methodology): “These are those predictors having forest averaged minimal depth lower that the mean of the minimal depth distribution, namely mean minimal depth threshold according to Ishwaran et al. (2010), here denoted by T*.”
A random forest guided tour (2015, background): “Random forests have been extended to the survival context by Ishwaran et al. (2008), who prove consistency of Random Survival Forests (RSF) algorithm assuming that all variables are factors. Yang et al. (2010) showed that by incorporating kernel functions into RSF, their algorithm KIRSF achieves bet”
Correlation and variable importance in random forests (2013, background): “Moreover, the impact of the correlations on other importance measures (see van der Laan [38], Ishwaran [18]) is a general question of great interest.”
Nonparametric variable importance assessment using machine learning techniques (2020, background): “The former category includes, for example, variable importance measures for random forests (Breiman, 2001; Ishwaran, 2007; Strobl et al. , 2007; Grömping, 2009) and neural networks (see, eg, Olden et al. , 2004), and ANOVA in linear models.”
Variable Importance Assessment in Regression: Linear Regression versus Random Forest (2009, methodology): “Theoretical results on random forests (Breiman 2001; Ishwaran 2007) are asymptotic in the number of trees, and a large number of trees has been reported to be particularly important when interest is in diagnostic quantities like variable importance (cf.”
High-Dimensional Variable Selection for Survival Data (2010, methodology): “A more effective method, which we use throughout this article, is to assign terminal node membership by random node assignment (Ishwaran 2007; Ishwaran et al. 2008).”
Variable Importance and Prediction Methods for Longitudinal Problems with Missing Variables (2015, neutral): “On the other hand, consider the case of regression and classification trees (e.g., random forests), where the VIM for a variable X is defined as the difference between prediction error when X is perturbed versus the prediction error otherwise [3,7].”