CAN ONE ESTIMATE THE UNCONDITIONAL DISTRIBUTION OF POST-MODEL-SELECTION ESTIMATORS?

By H. Leeb, Benedikt M. Poetscher. Econometric Theory, 2007

373 citations recorded by the source corpus.

We consider the problem of estimating the unconditional distribution of a post-model-selection estimator. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model-selection criterion such as the Akaike information criterion [AIC] or by a hypothesis testing procedure) and then estimating the parameters in the selected model (e.g., by least squares or maximum likelihood), all based on the same data set. We show that it is impossible to estimate the unconditional distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (local) minimax lower bounds on the performance of estimators for the distribution; performance is here measured by the probability that the estimation error exceeds a given threshold. These lower bounds are shown to approach ½ or even 1 in large samples, depending on the situation considered. Similar impossibility results are also obtained for the distribution of linear functions (e.g., predictors) of the post-model-selection estimator.The research of the first author was supported by the Max Kade Foundation and by the Austrian National Science Foundation (FWF), Grant no. P13868-MAT. A preliminary draft of the material in this paper was written in 1999.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/a56b8c46cbc273ba566c16e14e86130e9b194c86

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

PANEL UNIT ROOT TESTS WITH CROSS-SECTION DEPENDENCE: A FURTHER INVESTIGATION (2009, methodology): “As Leeb and Potscher (2008) showed, the asymptotic distribution can be a poor approximation for the finite-sample distributions for certain DGPs.”
Focused Information Criteria, Model Selection, and Model Averaging in a Tobit Model With a Nonzero Threshold (2012, background): “Another important motivation behind model averaging is that it provides a kind of insurance against selecting a very poor model, and thus holds promise for reducing the risk in regression estimation (Leung and Barron 2006).”
Machine Learning: An Applied Econometric Approach (2017, background): “In fact, Leeb and Pötscher (2006, 2008) develop conditions under which it is impossible to obtain (uniformly) consistent estimates of the distribution of model parameters after data-driven selection.”
Valid post-selection inference (2013, neutral): “Selection procedures are now maps M̂ :Y 7→ M̂(Y), Rn →M.(4.7) The following are examples of model universes with practical relevance; see also Leeb and Pötscher (2008a), Section 1.1, Example 1.”
Confidence intervals for low dimensional parameters in high dimensional linear models (2011, methodology): “…many non-zero coefficients of small or moderate magnitude, construction of a confidence interval for such a preconceived parameter is an important problem in and of itself and was open before our paper (Leeb and Potscher, 2006), but the method proposed is not limited to this application.”
Forecasting with Factor-Augmented Regression: A Frequentist Model Averaging Approach, Second Version (2013, neutral): “Finally, interval forecast based on model averaging is an important but challenging topic (Leeb and Pötscher, 2003, 2008).”
INFERENCE AFTER MODEL AVERAGING IN LINEAR REGRESSION MODELS (2018, background): “There is a large literature on inference after model selection, including Pötscher (1991), Kabaila (1995, 1998), Pötscher and Leeb (2009), and Leeb and Pötscher (2003, 2005, 2006, 2008, 2017).”
On the uniform asymptotic validity of subsampling and the bootstrap (2012, background): “See also Kabaila (1995), Pötscher (2002), Leeb and Pötscher (2006a, 2006b), Pötscher (2009) for related results in more complicated settings, including post-model selection, shrinkage-estimators and ill-posed problems.”
Uniform asymptotic inference and the bootstrap after model selection (2015, methodology): “In a sequence of papers, Leeb & Potscher (2003, 2006, 2008) prove that in a classical regression setting, it is impossible to find the distribution of a post-selection estimator of the underlying coefficients, even asymptotically.”
Sensitivity analysis using approximate moment condition models (2018, background): “17 in Lehmann and Romano (2005) gives the Jacobian with respect to t as Σ in the case where g(wi, θ ∗) is bounded, and the Jacobian with respect to θ is equal to Γ by definition.”
Weighted‐Average Least Squares (WALS): A Survey (2016, methodology): “Problems associated with inference after model selection have been investigated in Magnus (1999, 2002), Danilov and Magnus (2004a, b), Leeb and Pötscher (2003, 2005, 2006, 2008), and others.”
Valid Post-Selection Inference in High-Dimensional Approximately Sparse Quantile Regression Models (2013, background): “This is the so-called uniformity property or honesty property of the confidence regions (see, e.g., Leeb and Pötscher 2006; Romano and Wolf 2007; Romano and Shaikh 2012).”