Estimation and Accuracy after Model Selection

By Bradley Efron. 2013

356 citations recorded by the source corpus.

Classical statistical theory ignores model selection in assessing estimation accuracy. Here we consider bootstrap methods for computing standard errors and confidence intervals that take model selection into account. The methodology involves bagging, also known as bootstrap smoothing, to tame the erratic discontinuities of selection-based estimators. A useful new formula for the accuracy of bagging then provides standard errors for the smoothed estimators. Two examples, nonparametric and parametric, are carried through in detail: a regression model where the choice of degree (linear, quadratic, cubic, …) is determined by the Cp criterion and a Lasso-based estimation problem.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Metalearners for estimating heterogeneous treatment effects using machine learning (2017, background): “[36] Efron B (2014) Estimation and accuracy after model selection.”
Generalized random forests (2016, methodology): “Our asymptotic theory relates to an extensive recent literature on the statistics of random forests, most of which focuses on the regression case (Arlot and Genuer, 2014; Biau, 2012; Biau et al., 2008; Biau and Scornet, 2016; Breiman, 2004; Bühlmann and Yu, 2002; Chipman et al., 2010; Denil et al., ”
Bayesian Workflow (2020, methodology): “This is an example of the problem of post-selection inference (Berk et al., 2013, Efron, 2013).”
Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects (2017, unknown): “Wager and Athey (2018) is notable for providing the first inferential theory for CATE estimators arising from a random forests representation, based on the infinitesimal jackknife (Efron, 2014; Wager et al., 2014).”
Regression Analysis for Microbiome Compositional Data (2016, methodology): “racterizing the distribution of a post-selection estimator conditioned on the selection event. It is interesting to extend their approach to the high-dimensional regression problems with constraints. Efron (2014) developed a bootstrap smoothing procedure for computing the standard errors and condenc”
Causal Interaction in Factorial Experiments: Application to Conjoint Analysis (2018, unknown): “In particular, we estimate the selection probability for each AME and AMIE using one minus the proportion of 5000 bootstrap replicates in which all coe ffi cients for the corresponding factor or factor interaction are estimated to be zero (Efron 2014; Hastie, Tibshirani, and Wainwright 2015).”
Asymptotic post‐selection inference for the Akaike information criterion (2018, methodology): “Efron (2014) proposed using a bagging, i.e., bootstrap aggregating, estimator and derived its variance, using normal quantiles to obtain confidence intervals.”
Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife (2013, methodology): “The results presented here build on the jackknife-after-bootstrap methodology introduced by Efron (1992) and on the infinitesimal jackknife for bagging (IJ) (Efron, 2013).”
Bootstrapping and sample splitting for high-dimensional, assumption-lean inference (2016, methodology): “(2015) on random X and model misspecification; methods based on resampling or sample splitting (Chatterjee and Lahiri, 2011, 2013; Efron, 2014; Wasserman and Roeder, 2009; Meinshausen et al., 2009); stability selection (Meinshausen and Bühlmann, 2010; Shah and Samworth, 2013); the conformal inferenc”
Specification curve analysis (2020, background): “(5) Repeat steps 3 and 4 a large number of times (for example, 500 or 1,000).”
Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications (2015, unknown): “One proposal is that researchers modify the estimates of a given model to take into account an initial model selection process guided by fit (e.g., when deciding between a quadratic vs cubic polynomial (Efron, 2014)).”
Asymptotic Theory for Random Forests (2014, methodology): “Let (cid:98) V IJ ( x ; , Z 1 , ..., Z n ) be the inﬁnitesimal jackknife for random forests [14] as deﬁned in (4) .”
Novel seasonal land cover associations for eastern North American forest birds identified through dynamic species distribution modelling (2016, background): “Combining estimates across the ensemble controls for inter-model variability (Efron, 2014) and adapts to non-stationary in predictor–response relationships (Fink et al.”
Accurate and Precise Prediction of Soil Properties from a Large Mid-Infrared Spectral Library (2019, background): “In contrast, rather than remove one observation at a time, the infinitesimal jackknife approach assigns a weight to each observation to provide the estimate variance [73].”
Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects (with Discussion) (2020, unknown): “Wager and Athey (2018) is notable for providing the first inferential theory for CATE estimators arising from a random forests representation, based on the infinitesimal jackknife (Efron, 2014; Wager et al., 2014).”
Why Propensity Scores Should Not Be Used for Matching (2019, background): “At a minimum, model dependence creates additional o en unaccounted for uncertainty (King and Zeng 2007; Athey and Imbens 2015; Efron 2014).”
Conformal inference of counterfactuals and individual treatment effects (2020, methodology): “…with three competing methods offering qualitatively different approaches to uncertainty quantification as well as well- written R packages: • Causal Forest (Wager & Athey, 2018) uses the infinitesimal jackknife (Efron, 2014; Wager et al., 2014) to estimate the variance of CATE estimators.”
Quantifying Uncertainty in Random Forests via Confidence Intervals and Hypothesis Tests (2014, methodology): “Recently, Wager et al. (2014) proposed applying the jackknife and inﬁnitesimal jackknife procedures introduced by Efron (2014) for estimating standard errors in random forest predictions.”