Selective Sequential Model Selection

By William Fithian, Jonathan E. Taylor, Robert Tibshirani. 2015

66 citations recorded by the source corpus.

Many model selection algorithms produce a path of fits specifying a sequence of increasingly complex models. Given such a sequence and the data used to produce them, we consider the problem of choosing the least complex model that is not falsified by the data. Extending the selected-model tests of Fithian et al. (2014), we construct p-values for each step in the path which account for the adaptive selection of the model path using the data. In the case of linear regression, we propose two specific tests, the max-t test for forward stepwise regression (generalizing a proposal of Buja and Brown (2014)), and the next-entry test for the lasso. These tests improve on the power of the saturated-model test of Tibshirani et al. (2014), sometimes dramatically. In addition, our framework extends beyond linear regression to a much more general class of parametric and nonparametric model selection problems. To select a model, we can feed our single-step p-values as inputs into sequential stopping rules such as those proposed by G'Sell et al. (2013) and Li and Barber (2015), achieving control of the familywise error rate or false discovery rate (FDR) as desired. The FDR-controlling rules require the null p-values to be independent of each other and of the non-null p-values, a condition not satisfied by the saturated-model p-values of Tibshirani et al. (2014). We derive intuitive and general sufficient conditions for independence, and show that our proposed constructions yield independent p-values.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3ba6d46203af2f81f6414a940fdff38e3a6dc32b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Selecting the number of principal components: estimation of the true rank of a noisy matrix (2014, unknown): “This independence relation between p-values is shown in Theorem 2.3, which corresponds to a special case of Theorem 4 by Fithian et al. (2015), page 15.”
Bayesian Workflow (2020, background): “This “double dipping” can in principle threaten the frequency properties of our inferences, and it is important to be aware of the possibility of overfitting arising from model selection, as considered for example by Fithian et al. (2015) and Loftus (2015). A related issue is the garden of forking p”
More powerful post-selection inference, with application to the Lasso (2018, background): “Such a simple result is hinted at by Fithian et al. (2015) who derived truncation regions of this form in the context of sequential testing.”
Exact Post-Selection Inference for Sequential Regression Procedures (2014, methodology): “We computed the p-values from this article, called “saturated model p-values” in Fithian et al. (2015), and those called “selective model p-values” in the same article.”
Selective inference with a randomized response (2015, background): “Many examples of nonrandomized selective inference can be expressed in this way [cf. Fithian et al. (2015), Lee et al. (2016), Tibshirani et al. (2016)].”
Exact post-selection inference for the generalized lasso path (2018, methodology): “In the language of Fithian et al. [10, 11], the development of post-selection tests in this paper was done under a “saturated model” for the mean parameter θ —this treats θ as an arbitrary vector in R n , and the hypotheses being tested are all phrased in terms of certain linear contrasts of the…”