Ensemble Trees and CLTs: Statistical Inference for Supervised Learning

By Lucas K. Mentch, Giles Hooker. 2014

34 citations recorded by the source corpus.

This work develops formal statistical inference procedures for machine learning ensemble methods. Ensemble methods based on bootstrapping, such as bagging and random forests, have improved the predictive accuracy of individual trees, but fail to provide a framework in which distributional results can be easily determined. Instead of aggregating full bootstrap samples, we consider predicting by averaging over trees built on subsamples of the training set and demonstrate that the resulting estimator takes the form of a U-statistic. As such, predictions for individual feature vectors are asymptotically normal, allowing for confidence intervals to accompany predictions. In practice, a subset of subsamples is used for computational speed; here our estimators take the form of incomplete U-statistics and equivalent results are derived. We further demonstrate that this setup provides a framework for testing the significance of features. Moreover, the internal estimation method we develop allows us to estimate the variance parameters and perform these inference procedures at no additional computational cost. Simulations and illustrations on a real dataset are provided.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Estimation and Inference of Heterogeneous Treatment Effects using Random Forests (2015, methodology): “Meanwhile, their sampling variability has been analyzed by Duan [2011], Lin and Jeon [2006], Mentch and Hooker [2014], Sexton and Laake [2009], and Wager et al. [2014].”
Consistency of Random Forests (2014, methodology): “The proof of Theorem 1 is based on ideas developed by Nobel (1996), and worked out in Theorem 10.”
Uniform Convergence of Random Forests via Adaptive Concentration (2015, background): “Methods for doing statistical inference on random forest predictions are also available [31, 38, 43, 44].”
Asymptotic Theory for Random Forests (2014, methodology): “Mentch and Hooker [27] study the impact of Monte Carlo noise on subsampled estimators; the Monte Carlo distribution of V̂IJ is discussed by Wager et al.”
On the asymptotics of random forests (2014, background): “However, surprisingly, no study aims at clarifying the link between finite forests (finite M) and infinite forests (M = ∞) even if some authors (Mentch and Hooker, 2014; Wager et al., 2014) proved results on finite forest predictions at a fixed point x.”
A random forest guided tour (2015, methodology): “he practical situation. Recent attempts towards narrowing the gap between theory and practice include that ofDenil et al.(2013), who prove the consistency of a particular online forest,Wager(2014) andMentch and Hooker (2015), who study the asymptotic distribution of forests, andScornet et al. (2015)”