Correlation and variable importance in random forests

By Baptiste Gregorutti, Bertrand Michel, Philippe Saint-Pierre. 2013

996 citations recorded by the source corpus.

This paper is about variable selection with the random forests algorithm in presence of correlated predictors. In high-dimensional regression or classification frameworks, variable selection is a difficult task, that becomes even more challenging in the presence of highly correlated predictors. Firstly we provide a theoretical study of the permutation importance measure for an additive regression model. This allows us to describe how the correlation between predictors impacts the permutation importance. Our results motivate the use of the recursive feature elimination (RFE) algorithm for variable selection in this context. This algorithm recursively eliminates the variables using permutation importance measure as a ranking criterion. Next various simulation experiments illustrate the efficiency of the RFE algorithm for selecting a small number of variables together with a good prediction error. Finally, this selection algorithm is tested on the Landsat Satellite data from the UCI Machine Learning Repository.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Information Systems

In Molecular Biology

Citations in the corpus

Visualizing the Feature Importance for Black Box Models (2018, background): “ the approximation PFI[ S;approx = 1 n Sm P n i =1 P m k L(f^(x(˝ (i) k );x(i) C );y (i)) (L(f^(x(i));yi)) : (8) Eq. (8) is identical to the permutation importance of random forests formalized in [12] if we consider mas the number of trees, replace nwith the number of out-of-bag samples per tree a”
All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously (2018, methodology): “As in Gregorutti et al. (2017), both results in Proposition 15 readily generalize to additive models of the form fg1,g2(X1, X2) := g1(X1)+g2(X2), since permuting X1 is equivalent to permuting g1(X1).”
Machine Learning Predicts Laboratory Earthquakes (2017, background): “The most useful features are then selected recursively (Gregorutti et al., 2017).”
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, background): “followed by a rich line of research on the topic [4, 34, 56, 57, 131, 156].”
Fake News Detection Using Machine Learning Ensemble Methods (2020, result): “RF consists of large number of decision trees working individually to predict an outcome of a class where the ﬁnal prediction is based on a class that received majority votes. error rate is low in random forest as compared to other models, due to low correlation among trees [33].”
DIANet: Dense-and-Implicit Attention Network (2019, methodology): “Firstly, the layers are connected by the DIA-LSTM unit in DIANet and we use the random forest model (Gregorutti, Michel, and Saint-Pierre 2017) to visualize how the current layer depends on the preceding layers.”
Grouped variable importance with random forests and application to multiple functional data analysis (2014, methodology): “…conditional distribution of Y over X is normal and the conditional mean f is a linear function: f ( x ) = p + q j =1 α j x j with α = ( α 1 , . . . , α p , 0 , . . . 0) (cid:62) a sequence of deterministic coeﬃcients (see for instance Rao (1973), p. 522 and Section 3 in Gregorutti et al. (2014)).”
All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance (2018, methodology): “Gregorutti et al. (2017) show an equivalent formulation of Eq 3.4 under the stronger assumptions that fβ is equal to the conditional expectation function of Y (that is, fβ(x) = E(Y”
VSURF: An R Package for Variable Selection Using Random Forests (2015, positive): “In a recent paper Gregorutti et al. (2013) theoretically state that, in the case of additive models, irrelevant variables have null “theoretical” VI.”
Machine learning pipeline for battery state-of-health estimation (2021, positive): “Recursive feature elimination generally outperforms other conventional methods [45], [46], hence the adoption here (refer to Methods section for further details).”