Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
By Ery Arias-Castro, E.J. Candes, Yaniv Plan. 2010
223 citations recorded by the source corpus.
Testing for the significance of a subset of regression coefficients in a linear model, a staple of statistical analysis, goes back at least to the work of Fisher who introduced the analysis of variance (ANOVA). We study this problem under the assumption that the coefficient vector is sparse, a common situation in modern high-dimensional settings. Suppose we have p covariates and that under the alternative, the response only depends upon the order of p^(1−α) of those, 0 ≤ α ≤ 1. Under moderate sparsity levels, that is, 0 ≤ α ≤ 1/2, we show that ANOVA is essentially optimal under some conditions on the design. This is no longer the case under strong sparsity constraints, that is, α > 1/2. In such settings, a multiple comparison procedure is often preferred and we establish its optimality when α ≥ 3/4. However, these two very popular methods are suboptimal, and sometimes powerless, under moderately strong sparsity where 1/2 1/2. This optimality property is true for a variety of designs, including the classical (balanced) multi-way designs and more modern “p > n” designs arising in genetics and signal processing. In addition to the standard fixed effects model, we establish similar results for a random effects model where the nonzero coefficients of the regression vector are normally distributed.
Subfields: Computational Mechanics, Management Science and Operations Research, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/3801a467e7876d4b4d0fe26c1cba6a46f9002a66
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computational Mechanics
In Management Science and Operations Research
In Statistics and Probability
- Jing Lei
- Adam D. Smith
- E.J. Candes
- Yu-Xiang Wang
- Cynthia Dwork
- Abhradeep Thakurta
- Mark Bun
- Robert Tibshirani
Citations in the corpus
- Adaptive Global Testing for Functional Linear Models (2014, background): “Such a framework (also known as the detection boundary problem) has been studied in highdimensional and nonparametric testing problems such as Ingster (1982), Donoho and Jin (2004), Ingster et al. (2010), Ingster et al. (2012), Arias-Castro et al. (2011).”