Analysis of a Random Forests Model

By Gérard Biau. 2010

1,647 citations recorded by the source corpus.

Random forests are a scheme proposed by Leo Breiman in the 2000's for building a predictor ensemble with a set of decision trees that grow in randomly selected subspaces of data. Despite growing interest and practical use, there has been little exploration of the statistical properties of random forests, and little is known about the mathematical forces driving the algorithm. In this paper, we offer an in-depth analysis of a random forests model suggested by Breiman (2004), which is very close to the original algorithm. We show in particular that the procedure is consistent and adapts to sparsity, in the sense that its rate of convergence depends only on the number of strong features and not on how many noise variables are present.

Subfields: Artificial Intelligence, Information Systems, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Statistical Learning Theory

Citations in the corpus

Scaling Laws for Neural Language Models (2020, neutral): “Power-law scalings with model and dataset size in density estimation [Was06] and in random forest models [Bia12] may be connected with our results.”
A Neural Scaling Law from the Dimension of the Data Manifold (2020, neutral): “Our work was also partly inspired by similar scaling relations in random forest models; with some added assumptions, it is possible to prove them [Bia12].”
AutoScore: A Machine Learning–Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records (2020, background): “Each tree sees only a subset of variables and part of the observations by resampling, which guarantees that the trees are decorrelated and, therefore, less prone to overfitting [19].”
Local Linear Forests (2018, methodology): “Bühlmann and Yu (2002) showed how bagging makes forests smoother than single trees, while Biau (2012) and Scornet, Biau, and Vert (2015) established asymptotic risk consistency of random forests under specific assumptions.”
Metalearners for estimating heterogeneous treatment effects using machine learning (2017, background): “For example, Athey and Imbens (2016, 2015) develop Classification and Regression Trees (CART) for CATE estimation, and Wager and Athey (2015) provide asymptotically valid inference methods for causal forests—a variant of Random Forests that is particularly amenable to theoretical analysis because it”
Estimation and Inference of Heterogeneous Treatment Effects using Random Forests (2015, positive): “To our knowledge, all existing results on pointwise consistency of regression forests [e.g., Meinshausen, 2006, Biau, 2012] require an analogous condition on E Y X = x .”
Uniform Convergence of Random Forests via Adaptive Concentration (2015, background): “The best existing convergence results for random forests either only provide asymptotic consistency guarantees without rates of convergence [36], or assume a substantially simpliﬁed training procedure where tree splits are chosen without looking at the training data [4].”
Towards optimal doubly robust estimation of heterogeneous causal effects (2020, background): “…are a fundamental class of regression estimators and include, for example, linear regression, series methods, local polynomials, nearest neighbor matching, smoothing splines, kernel ridge regression, and some versions of random forests [Biau, 2012, Scornet, 2016, Verdinelli and Wasserman, 2021].”
The Five Factor Model of personality and evaluation of drug consumption risk (2015, methodology): “Random forest was used for building a predictor ensemble with a set of decision trees that grow in randomly selected subspaces of data [4].”
Application of machine learning to construction injury prediction (2016, methodology): “RF was selected because it stands among the most accurate general-purpose classi ﬁ ers to date [5], and has shown to be effective in a variety of other ﬁ elds.”
A Parallel Random Forest Algorithm for Big Data in a Spark Cloud Computing Environment (2017, background): “The rest of the paper is organized as follows.”
Functional random forest with applications in dose-response predictions (2019, background): “Assume that γ Θ x ( , )  be the partition containing x , the tree response takes the following form 18,20,21 with corresponding weights i i tr tr Let the T trees of RF be denoted by , , , T 1 2 Θ Θ … Θ and w i ( x ) to be the average weights over the forest.”
Asymptotic Theory for Random Forests (2014, background): “Most existing theoretical results about random forests aim to establish the consistency of random forest predictions [2, 3, 7, 26].”
Ensemble Trees and CLTs: Statistical Inference for Supervised Learning (2014, neutral): “The second was initially considered by Friedman [1991] in development of the Multivariate Adaptive Regression Spline (MARS) procedure and was recently investigated by Biau [2012].”
Supervised Neighborhoods for Distributed Nonparametric Regression (2016, methodology): “Most notably, our analysis relies on the assumption that the procedure used to build the regression trees proceeds independently of the data used in building the local models; this assumption is also made in [21] and is similar to the ‘honest tree’ assumption defined in [22].”
Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests (2014, methodology): “Biau (2012) further examines the consistency of random forests and investigates their behavior in the presence of a sparse feature space.”
Analysis of purely random forests bias (2014, background): “Breiman first began to study such simplified RF (Breiman, 2000), and then well-established results were obtained by Biau et al. (2008). The main difference between PRF and RF is that, in PRF, partitioning of the input sylvain.”
Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife (2013, background): “Several variants of random forests have been analyzed theoretically by, e.g., Biau et al. (2008), Biau (2012), Lin and Jeon (2006), and Meinshausen (2006).”