A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data

By Bjoern H Menze, B. Michael Kelm, Ralf Masuch, Uwe Himmelreich, Peter Bachert, Wolfgang Petrich, Fred A. Hamprecht. BMC Bioinformatics, 2009

1,177 citations recorded by the source corpus.

BackgroundRegularized regression methods such as principal component or partial least squares regression perform well in learning tasks on high dimensional spectral data, but cannot explicitly eliminate irrelevant features. The random forest classifier with its associated Gini feature importance, on the other hand, allows for an explicit feature elimination, but may not be optimally adapted to spectral data due to the topology of its constituent classification trees which are based on orthogonal splits in feature space.ResultsWe propose to combine the best of both approaches, and evaluated the joint use of a feature selection based on a recursive feature elimination using the Gini importance of random forests' together with regularized classification methods on spectral data sets from medical diagnostics, chemotaxonomy, biomedical analytics, food science, and synthetically modified spectral data. Here, a feature selection using the Gini feature importance with a regularized classification by discriminant partial least squares regression performed as well as or better than a filtering according to different univariate statistical tests, or using regression coefficients in a backward feature elimination. It outperformed the direct application of the random forest classifier, or the direct application of the regularized classifiers on the full set of features.ConclusionThe Gini importance of the random forest provided superior means for measuring feature relevance on spectral data, but – on an optimal subset of features – the regularized classifiers might be preferable over the random forest classifier, in spite of their limitation to model linear dependencies only. A feature selection based on Gini importance, however, may precede a regularized linear classification to identify this optimal subset of features, and to earn a double benefit of both dimensionality reduction and the elimination of noise from the classification task.

Subfields: Analytical Chemistry, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/418fa45ba1cb0541be4ca1f671cf14fc5e132426

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Statistics and Probability

Citations in the corpus

Relevant Feature Set Estimation with a Knock-out Strategy and Random Forests (2015, positive): “…of techniques borrowing ideas from themachine learning literature (DeMartino et al., 2008), (Mourao-Miranda et al., 2005; Kriegeskorte et al., 2006; Menze et al., 2009; Sabuncu and Van Leemput, 2012; Langs et al., 2011; Yamashita et al., 2008; Gaonkar and Davatzikos, 2012; Rondina et al., 2014)…”
Multi-method ensemble selection of spectral bands related to leaf biochemistry (2015, positive): “This pronounced and almost binary classification into important/ non-important bands is considered a major advantage of the RF-based band selection compared to other algorithms (Menze et al., 2009).”
On Oblique Random Forests (2011, background): “The Gini feature importance is obtained from an ensemble of 1500 trees, and the oRF feature importance using an ensemble with 9000 trees.”
A tutorial review: Metabolomics and partial least squares-discriminant analysis--a marriage of convenience or a shotgun wedding. (2015, background): “authors emphasize that RF may not always be adapted to spectral data due to ‘the topology of its constituent classification trees’ [138]; however, they appreciate the capabilities of RF in terms of feature selection.”
Random Forest ensembles for detection and prediction of Alzheimer's disease with a good between-cohort robustness (2014, background): “…techniques in terms of ability to handle highly non-linear biological data, robustness to noise, tuning simplicity (compared to other ensemble learning algorithms) and opportunity for ef ﬁ cient parallel processing (De Bruyn et al., 2013; Caruana and Niculescu-Mizil, 2006; Menze et al., 2009).”
Testing the reliability and stability of the internal accuracy assessment of random forest for classifying tree defoliation levels using different validation methods (2015, positive): “…Lawrence et al. (2006), Adam et al. (2011), Breiman and Cutler (2012) and Waske et al. (2007) have supported this claim, others such as Bylander (2002), Ruiz-Gazen and Villa (2007), (Statnikov et al. 2008), Strobl and Zeileis (2008) and Menze et al. (2009) have contested the validity of this claim.”
Analysis of Machine Learning Techniques for Heart Failure Readmissions (2016, background): “RF involves the creation of multiple decision trees(16) that sort and identify important variables for prediction.(14,17) Boosting algorithms harness the power of weaker predictors by creating combined and weighted predictive variables.”
Monitoring land cover change in urban and peri-urban areas using dense time stacks of Landsat satellite data and a data mining approach (2012, background): “RF have been favored over principal components analysis due to their explicit estimation of variable importance, as well as their increased flexibility with large datasets, speed in processing, and lack of assumptions on data distribution (Menze et al., 2009).”
Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging (2013, methodology): “The random decision forest (RDF) method (Breiman, 2001; Menze et al., 2009) has also been employed to segment MS lesions (Geremia et al., 2011) using two types of features simultaneously: local and context-rich features.”
Relief-Based Feature Selection: Introduction and Review (2017, methodology): “Alternatively in a modeling algorithm like a random forest, estimated feature importance scores can be applied to select a feature subset (Menze et al., 2009).”
In Silico Predictive Modeling of CRISPR/Cas9 guide efficiency (2015, background): “This measure has a close, empirical correspondence with the importance of the feature that would be obtained with a permutation test, and can also be viewed as a relative decrease in entropy provided by splitting on that feature 20,31 .”
Nanomedicine-Driven Modulation of the Gut–Brain Axis: Innovative Approaches to Managing Chronic Inflammation in Alzheimer’s and Parkinson’s Disease (2025, unknown): “The additional feature of RF is its intrinsic feature selection that assigns an importance score to features, reducing the variable space [151].”