Analysis of purely random forests bias

By Sylvain Arlot, Robin Genuer. 2014

81 citations recorded by the source corpus.

Random forests are a very effective and commonly used statistical method, but their full theoretical analysis is still an open problem. As a first step, simplified models such as purely random forests have been introduced, in order to shed light on the good performance of random forests. In this paper, we study the approximation error (the bias) of some purely random forest models in a regression framework, focusing in particular on the influence of the number of trees in the forest. Under some regularity assumptions on the regression function, we show that the bias of an infinite forest decreases at a faster rate (with respect to the size of each tree) than a single tree. As a consequence, infinite forests attain a strictly better risk rate (with respect to the sample size) than single trees. Furthermore, our results allow to derive a minimum number of trees sufficient to reach the same rate as an infinite forest. As a by-product of our analysis, we also show a link between the bias of purely random forests and the bias of some kernel estimators.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Generalized random forests (2016, methodology): “Our asymptotic theory relates to an extensive recent literature on the statistics of random forests, most of which focuses on the regression case (Arlot and Genuer, 2014; Biau, 2012; Biau, Devroye and Lugosi, 2008; Biau and Scornet, 2016; Bühlmann and Yu, 2002; Denil, Matheson and De Freitas,…”
Uniform Convergence of Random Forests via Adaptive Concentration (2015, background): “Ever since their introduction by Breiman [8], many authors have studied the theoretical properties of random forests in order to explain their performance [1, 4, 5, 9, 16, 28, 30, 35, 36, 37, 43].”
Adaptive Concentration of Regression Trees, with Application to Random Forests (2015, background): “Several authors have studied the theoretical properties of trees and forests in order to explain their performance [1, 5, 6, 9, 14, 19, 28, 31, 33, 37, 38, 39, 45].”
Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution (2018, methodology): “(2008); Biau (2012); Arlot and Genuer (2014) that, in particular, generate a connection to kernel methods observed in Davies and Ghahramani (2014) and Scornet (2016). See Section 3.1 of Biau and Scornet (2016) for an overview of these developments.”
A random forest guided tour (2015, background): “The most recent advances in this direction are by Arlot and Genuer (2014), who show that a simplified forest model can be written as a kernel estimate, and provide its rates of convergence.”