Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife

By Stefan Wager, Trevor J. Hastie, Bradley Efron. 2013

434 citations recorded by the source corpus.

We study the variability of predictions made by bagged learners and random forests, and show how to estimate standard errors for these methods. Our work builds on variance estimates for bagging proposed by Efron (1992, 2013) that are based on the jackknife and the infinitesimal jackknife (IJ). In practice, bagged predictors are computed using a finite number B of bootstrap replicates, and working with a large B can be computationally expensive. Direct applications of jackknife and IJ estimators to bagging require B = Θ(n1.5) bootstrap replicates to converge, where n is the size of the training set. We propose improved versions that only require B = Θ(n) replicates. Moreover, we show that the IJ estimator requires 1.7 times less bootstrap replicates than the jackknife to achieve a given accuracy. Finally, we study the sampling distributions of the jackknife and IJ variance estimates themselves. We illustrate our findings with multiple experiments and simulation studies.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Local Linear Forests (2018, methodology): “Early versions of confidence intervals for random forests, backed by heuristic arguments and empirical evidence, were proposed by Sexton and Laake (2009) and Wager, Hastie, and Efron (2014).”
Consistency of Random Forests (2014, background): “It is one of the most effective computationally intensive procedures to improve on unstable estimates, especially for large, high-dimensional data sets where finding a good model in one step is impossible because of the complexity and scale of the problem (Bühlmann and Yu, 2002; Kleiner et al., 2014”
Uniform Convergence of Random Forests via Adaptive Concentration (2015, background): “[44] Stefan Wager, Trevor Hastie, and Bradley Efron.”
FairTest: Discovering Unwarranted Associations in Data-Driven Applications (2015, background): “To estimate the variance in a patient’s target value, we train multiple predictors over random data subsets, and infer prediction intervals for our algorithm’s outputs [57].”
Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests (2014, methodology): “Recently, Wager et al. (2014) proposed applying the jackknife and inﬁnitesimal jackknife procedures introduced by Efron (2014) for estimating standard errors in random forest predictions.”
Asymptotic Theory for Random Forests (2014, methodology): “[33], who showed empirically that the method worked well for many problems of interest.”
Accurate and Precise Prediction of Soil Properties from a Large Mid-Infrared Spectral Library (2019, methodology): “The infinitesimal jackknife approach has been considered to give more stable predictions compared with the jackknife approach [72].”
A comparison of some conformal quantile regression methods (2019, background): “There exist a vast selection of statistical and machine learning algorithms that can provide approximate answers to this question [3, 4].”
Random forest as a generic framework for predictive modeling of spatial and spatio-temporal variables (2018, methodology): “…of the predictions of random forest for regression-type problems can be estimated using211 several approaches:212 • The Jackknife-after-Bootstrap method (see e.g. Wager et al. (2014)).213 • The U-statistics approach of Mentch and Hooker (2016).214 • The Monte Carlo simulations (both target…”
Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects (2017, positive): “Wager and Athey (2018) is notable for providing the first inferential theory for CATE estimators arising from a random forests representation, based on the infinitesimal jackknife (Efron, 2014; Wager et al., 2014).”
Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution (2018, methodology): “In particular, the consistency of methods related to Random Forests (RFs: Breiman, 2001) has been demonstrated in Biau (2012); Scornet et al. (2015) while Wager et al. (2014); Mentch and Hooker (2016); Wager and Athey (2017) and Athey et al. (2016) prove central limit theorems for RF predictions.”
Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance (2020, background): “, 2019) as well as means by which standard errors (Sexton and Laake, 2009), confidence intervals (Wager et al., 2014; Mentch and Hooker, 2016), and hypothesis testing procedures (Mentch and Hooker, 2016, 2017; Coleman et al.”
High-Dimensional Materials and Process Optimization Using Data-Driven Experimental Design with Well-Calibrated Uncertainty Estimates (2017, methodology): “The sample-wise variance is defined as the average of the jackknife-after-bootstrap and infinitesimal jackknife variance estimates with a Monte Carlo sampling correction [17]:”