Valid post-selection inference

By Richard A. Berk, Lawrence D. Brown, Andreas Buja, Kai Zhang, Linda H. Zhao. 2013

651 citations recorded by the source corpus.

It is common practice in statistical data analysis to perform data-driven variable selection and derive statistical inference from the resulting model. Such inference enjoys none of the guarantees that classical statistical theory provides for tests and confidence intervals when the model has been chosen a priori. We propose to produce valid ``post-selection inference'' by reducing the problem to one of simultaneous inference and hence suitably widening conventional confidence and retention intervals. Simultaneity is required for all linear functions that arise as coefficient estimates in all submodels. By purchasing ``simultaneity insurance'' for all possible submodels, the resulting post-selection inference is rendered universally valid under all possible model selection procedures. This inference is therefore generally conservative for particular selection procedures, but it is always less conservative than full Scheffe protection. Importantly it does not depend on the truth of the selected submodel, and hence it produces valid inference even in wrong models. We describe the structure of the simultaneous inference problem and give some asymptotic results.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Forward stability and model path selection (2021, neutral): “Finally, we are not seeking to select a particular model and adjust variance estimates to account for additional uncertainty as might be done via some form of post-selection inference (e.g. Berk et al. 2013; Lee et al. 2016; Tibshi-rani et al. 2016; Taylor and Tibshirani 2018).”
Estimation and Inference of Heterogeneous Treatment Effects using Random Forests (2015, background): “In general, the challenge in using adaptive methods as the basis for valid statistical inference is that selection bias can be diﬃcult to quantify; see Berk et al. [2013], Chernozhukov et al. [2015], Taylor and Tibshirani [2015], and references therein for recent advances.”
Uniform Convergence of Random Forests via Adaptive Concentration (2015, methodology): “The guarantees provided by our method have a similar practical ﬂavor to those given by Berk et al. [3], in that both approaches protect against statistical instability without promising to recover the actual optimal conditional mean response function.”
Optimal Inference After Model Selection (2014, methodology): “For example, the “post-selection inference” (PoSI) method of Berk et al. (2013) constructs simultaneous (1 − α ) conﬁdence intervals for all parameters of all linear regression models that were ever under consideration.”
Post-Selection and Post-Regularization Inference in Linear Models with Many Controls and Instruments (2015, background): “See, for example, Berk et al. (2013), G’Sell et al. (2013), Lee and Taylor (2014), Lee et al. (2013), Lockhart et al. (2014), Loftus and Taylor (2014), Taylor et al. (2014), and Fithian et al.”
Beyond subjective and objective in statistics (2015, methodology): “This issue has recently attracted some research (for example, Berk et al., 2013), but this still requires a transparent listing of all the possible modeling decisions that could be made (V1b), which often is missing, and which may not even be desirable as long as the methods are used in an…”
Predicting the long-term stock market volatility: A GARCH-MIDAS model with variable selection (2020, background): “The variable selection can have a detrimental impact on the subsequently constructed inference procedures, such as confidence intervals, if these are constructed in a “naïve” way where the model selection is ignored (Berk et al., 2013; Leeb et al., 2015).”
Exact post-selection inference for the generalized lasso path (2018, neutral): “Berk et al. [3], Lockhart et al. [27] circumvent this by considering diﬀer-ent test statistics, rather than the standard studentized pivot (the standard for inference without selection).”
Assumption Lean Regression (2018, background): “Research is underway to provide valid post-selection inference (e.g., Berk et al. 2013, Lee et al. 2016), which is an important special case.”
An impact assessment of machine learning risk forecasts on parole board decisions and recidivism (2017, methodology): “Nevertheless, asymptotically valid statistical estimates can be obtained for an approximation of the average treatment effect (Berk et al. 2014a, b; Buja et al.2015).”
Models as Approximations II: A Model-Free Theory of Parametric Regression (2016, methodology): “In this case the regression functional resulting from (2) will be called a maximumlikelihood functional or ML functional for short.”
Uniformly valid confidence intervals post-model-selection (2016, methodology): “The present article is devised in the spirit of Berk et al. (2013), in the sense that we aim at inference post-model-selection that is valid irrespective of the employed model selection procedure.”
A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results (2018, methodology): “…a framework for speciﬁc variable selection procedures (forward stepwise regression, least angle regression, and the lasso regularization path) and Berk et al. (2013) present a framework that holds for all variable selection procedures that is more conservative than Scheffé protection (Scheffé…”
Using Regression Kernels to Forecast A Failure to Appear in Court (2014, background): “) (12)The recent literature is quite rich, but key problems are not yet solved (Berk et al., 2013; Lockhard et al., 2013; Voorman et al. 2014).”
Distribution-Free Predictive Inference for Regression (2016, background): “In the second class, post-selection approaches, the focus is on covering coefficients in the best linear approximation to μ given a subset of selected covariates (see, e.g., Berk et al. 2013; Fithian, Sun, and Taylor 2014; Tian and Taylor 2017, 2018; Lee et al. 2016; Tibshirani et al. 2016).”