Optimal Inference After Model Selection

By William Fithian, Dennis L. Sun, Jonathan E. Taylor. 2014

371 citations recorded by the source corpus.

To perform inference after model selection, we propose controlling the selective type I error; i.e., the error rate of a test given that it was performed. By doing so, we recover long-run frequency properties among selected hypotheses analogous to those that apply in the classical (non-adaptive) context. Our proposal is closely related to data splitting and has a similar intuitive justification, but is more powerful. Exploiting the classical theory of Lehmann

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Counterfactual Prediction with Deep Instrumental Variables Networks (2016, methodology): “Splitting procedures like this have a long history [Cox, 1975], and have recently regained popularity as an option for inference post model-selection [e.g., Fithian et al., 2015, Wager and Athey, 2015].”
Uniform Convergence of Random Forests via Adaptive Concentration (2015, methodology): “Another approach to post-selection inference is to ﬁx the model-selection method and then explicitly condition on the selection event [18, 26, 29, 40].”
A Minimax Theory for Adaptive Data Analysis (2016, methodology): “(2014); Fithian et al. (2014); Taylor & Tibshirani (2015) and others developed a series of “selective inference” methods that work with specific variable selection tools (e.”
Selective inference with a randomized response (2015, methodology): “(2014), Lee & Taylor (2014), Fithian et al. (2014). Loosely speaking, there are two stages in selective inference.”
Identification of and Correction for Publication Bias (2017, neutral): “For example, provided the density f X ⇤|⇥⇤(x|✓) belongs to an exponential family one can form confidence intervals by inverting uniformly most powerful unbiased tests as in Fithian et al. (2014).”
Asymptotics of Selective Inference (2015, background): “In Fithian et al. (2014), it is argued that the natural quantity to control is a selective version of Type I error (or a selective version of coverage for the case of intervals).”
Bootstrapping and sample splitting for high-dimensional, assumption-lean inference (2016, background): “This began with [41], but was developed more fully in [37], [29], and [68].”
Max-Information, Differential Privacy, and Post-selection Hypothesis Testing (2016, neutral): “L G ] 1 3 A pr 2 Contents”
Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach (2015, neutral): “Related developments within this approach appear in G’Sell et al. (2013), Lee and Taylor (2014), Lee et al. (2013), Lockhart et al. (2014), Loftus and Taylor (2014), Taylor et al. (2014), and Fithian et al. (2014).”
Exact post-selection inference for the generalized lasso path (2018, methodology): “Fithian et al. [10, 11] show how to also conduct tests under the “selected model”—to use the 1d fused lasso as an example, this would model the mean as a vector that is piecewise constant with breaks at the selected changepoints.”
How Much Does Your Data Exploration Overfit? Controlling Bias via Information Usage (2015, background): “One recent line of work [8], [9] proposes a framework for assigning significance and confidence intervals in selective inference, where model selection and significance testing are performed on the same dataset.”
Why adaptively collected data have negative bias and how to correct for it (2017, background): “Fithian et al. [9] discussed data splitting in post-selective inference.”
Distribution-Free Predictive Inference for Regression (2016, background): “In the second class, post-selection approaches, the focus is on covering coefficients in the best linear approximation to μ given a subset of selected covariates (see, e.g., Berk et al. 2013; Fithian, Sun, and Taylor 2014; Tian and Taylor 2017, 2018; Lee et al. 2016; Tibshirani et al. 2016).”
Valid confidence intervals for post-model-selection predictors (2014, background): “ed by Belloni et al. (2011, 2014), van de Geer et al. (2014), and Zhang and Zhang (2014). An alternative to the PoSI approach, that also focuses on (n) M^ as the quantity of interest, is developed in Fithian et al. (2015), Lee et al. (2015), Lee and Taylor (2014), Tian and Taylor (2015), and Tibshir”