Hyperparameters and tuning strategies for random forest

By Philipp Probst, Marvin N. Wright, Anne-Laure Boulesteix. WIREs Data Mining Knowl. Discov., 2018

2,058 citations recorded by the source corpus.

The random forest (RF) algorithm has several hyperparameters that have to be set by the user, for example, the number of observations drawn randomly for each tree and whether they are drawn with or without replacement, the number of variables drawn randomly for each split, the splitting rule, the minimum number of samples that a node must contain, and the number of trees. In this paper, we first provide a literature review on the parameters' influence on the prediction performance and on variable importance measures. It is well known that in most cases RF works reasonably well with the default values of the hyperparameters specified in software packages. Nevertheless, tuning the hyperparameters can improve the performance of RF. In the second part of this paper, after a presenting brief overview of tuning strategies, we demonstrate the application of one of the most established tuning strategies, model‐based optimization (MBO). To make it easier to use, we provide the tuneRanger R package that tunes RF with MBO automatically. In a benchmark study on several datasets, we compare the prediction performance and runtime of tuneRanger with other tuning implementations in R and RF with default hyperparameters.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/31a65d107bffcbf83811a11a580cafd3d3b9a504

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Random forest versus logistic regression: a large-scale benchmark experiment (2018, background): “However, noticeable improvements may be achieved in some cases [20].”
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, methodology): “However, it is often the case that some HPCs work relatively well on a large variety of data sets (Probst et al., 2019).”
Grid search in hyperparameter optimization of machine learning models for prediction of HIV/AIDS test results (2021, methodology): “…are employed such as Gradient Boosting (GB) [7], Support Vector Machine (SVM) [6], Extra Tree (ET) [8], K-nearest Neighbor (KNN) [9], Decision Tree (DT) [10], AdaBoost (AB) [11], Random Forest (RF) [12], and Logistic Regression (LR) [13] to test our methods efficiency on the considered dataset.”
Support Vector Machine Versus Random Forest for Remote Sensing Image Classification: A Meta-Analysis and Systematic Review (2020, methodology): “Another reason for this value being commonly used could be the fact that 500 is the default value in the software packages like R package; “randomForest” [128].”
Machine learning and soil sciences: a review aided by machine learning tools (2019, background): “For an indepth report of hyper-parameter tuning and its effects in the context of random forest, we refer the reader to Probst et al. (2019).”