Sparse estimators and the oracle property, or the return of Hodges’ estimator

By H. Leeb, Benedikt M. Poetscher. 2007

241 citations recorded by the source corpus.

We point out some pitfalls related to the concept of an oracle property as used in Fan and Li (2001, 2002, 2004) which are reminiscent of the well-known pitfalls related to Hodges’ estimator. The oracle property is often a consequence of sparsity of an estimator. We show that any estimator satisfying a sparsity property has maximal risk that converges to the supremum of the loss function; in particular, the maximal risk diverges to infinity when ever the loss function is unbounded. For ease of presentation the result is set in the framework of a linear regression model, but generalizes far beyond that setting. In a Monte Carlo study we also assess the extent of the problem infinite samples for the smoothly clipped absolute deviation (SCAD) estimator introduced in Fan and Li (2001). We find that this estimator can perform rather poorly infinite samples and that its worst-case performance relative to maximum likelihood deteriorates with increasing sample size when the estimator is tuned to sparsity.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/2ae80174da7fc699e851be2a538932b8cb0057e8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Unified LASSO Estimation by Least Squares Approximation (2007, result): “These results further confirm the theoretical findings of Leeb and Pötscher (2005) stating that no consistent model selection method including LSA can have a good minimax efficiency.”
Model Selection (2018, background): “As shown in Leeb and Pötscher (2008a), the unboundedness phenomenon (19) also applies to so-called ‘sparse’ estimators as considered, e.g., in Fan and Li (2001); cf. Section 4.”
On the distribution of the adaptive LASSO estimator (2008, unknown): “[This is in line with theoretical results in Leng et al. (2006), see also Leeb & P¨otscher (2008b).]”
Consistencies and rates of convergence of jump-penalized least squares estimators (2009, unknown): “In general, a side phenomenon related to such sparsity of an estimator is a bad uniform risk behavior [see Pötscher and Leeb (2008)].”
Post-Selection Inference for Generalized Linear Models With Many Controls (2013, background): “Leeb and Pötscher (2005), Leeb and Pötscher (2008), Pötscher (2009), and Pötscher and Leeb (2009) have shown that such traditional/naive inference approach is not robust to violations of the separation condition, which bounds the magnitude of the nonzero coefficients away from zero.”
GENERALIZED ADDITIVE PARTIAL LINEAR MODELS WITH HIGH-DIMENSIONAL COVARIATES (2013, result): “With regard to the consistency and asymptotic normality results presented previously, we also mention some recent works (Leeb and Pötscher, 2005, 2008) that have studied the (uniform) consistency of estimates of distribution functions of the penalized estimators.”
Efficient and Accurate Lp-Norm Multiple Kernel Learning (2009, result): “However, also the rate of convergence to the true estimator needs to be considered, thus we conjecture that the rate slower than √ n which is common to sparse estimators [11] may be one of the reasons for finding excellent (nonasymptotic) results in non-sparse MKL.”
Focused Information Criteria and Model Averaging for the Cox Hazard Regression Model (2006, methodology): “This disturbing max-risk phenomenon has been discussed by Foster and George (1994) for the BIC in multiple regression, by Yang (2005) for more general oracle procedures in a context of additive regression models, and by Leeb and Pötscher (2004, 2005).”
Valid post-selection inference (2013, unknown): “Selection procedures are now maps M̂ :Y 7→ M̂(Y), Rn →M.(4.7) The following are examples of model universes with practical relevance; see also Leeb and Pötscher (2008a), Section 1.1, Example 1.”
Confidence sets based on penalized maximum likelihood estimators in Gaussian regression (2008, background): “From (4) and (6) after rearranging terms we obtain for > bn”
Select the Valid and Relevant Moments: An Information-Based LASSO for GMM with Many Moments (2013, methodology): “This is related to the post model selection inference problem investigated by Leeb and Pötscher (2005, 2008), Andrews and Guggenberger (2009, 2010), Guggenberger (2010), Belloni, Chernozhukov, and Hansen (2011), and McCloskey (2012), among others.”