Narrowing the Gap: Random Forests In Theory and In Practice

By Misha Denil, David Matheson, Nando de Freitas. 2013

244 citations recorded by the source corpus.

Despite widespread interest and practical use, the theoretical properties of random forests are still not well understood. In this paper we contribute to this understanding in two ways. We present a new theoretically tractable variant of random regression forests and prove that our algorithm is consistent. We also provide an empirical evaluation, comparing our algorithm and other theoretically tractable random forest models to the random forest algorithm used in practice. Our experiments provide insight into the relative importance of different simplifications that theoreticians have made to obtain tractable models for analysis.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Generalized random forests (2016, methodology): “, 2015; Wager and Athey, 2017), and a sub-sample splitting technique that we call honesty (Biau, 2012; Denil et al., 2014; Wager and Athey, 2017).”
Understanding Random Forests: From Theory to Practice (2014, methodology): “Finally, let us complete this review by mentioning consistency results in the case of domain-specific adaptations of random forests, including quantile regression [Meinshausen, 2006], survival analysis [Ishwaran and Kogalur, 2010] and online forest construction [Denil et al., 2013b].”
A random forest guided tour (2015, background): “As observed by Denil et al. (2014), this state of affairs has led to polarization between theoretical and empirical contributions to the literature.”
Ensemble Trees and CLTs: Statistical Inference for Supervised Learning (2014, unknown): “Recently, Denil et al. [2013] proved consistency for a mathematically tractible variant of random forests and in some cases, achieved empirical performance on par with the original random forest procedure suggested by Breiman.”
Asymptotic Theory for Random Forests (2014, background): “[13] proposed a form of random forests that satisfy Definition 2 while matching the empirical performance of CART-based random forests on several datasets.”
Adaptive Concentration of Regression Trees, with Application to Random Forests (2015, background): “However, from the perspective of existing results, we have no particularly strong reasons to believe that forest predictions ought to be well behaved in high dimensions: The best existing convergence results for random forests either only provide fixed-dimensional asymptotic consistency guarantees [”
Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests (2014, background): “Recently, Denil et al. (2013) proved consistency for a mathematically tractable variant of random forests and in some cases, achieved empirical performance on par with the original random forest procedure suggested by Breiman.”
Machine Unlearning for Random Forests (2020, result): “There is also empirical evidence that bootstrapping does not improve predictive performance (Zaman & Hirose, 2009; Denil et al., 2014; Mentch & Hooker, 2016), which was consistent with our own experiments (Appendix: §B.2, Table 5).”
Heteroscedastic Treed Bayesian Optimisation (2014, background): “The tree is constructed in a recursive manner by choosing splits on features and thresholds so as to reduce uncertainty (Denil et al. , 2014).”