Higher criticism for detecting sparse heterogeneous mixtures

By David L. Donoho, Jiashun Jin. 2004

803 citations recorded by the source corpus.

Higher criticism, or second-level significance testing, is a multiplecomparisons concept mentioned in passing by Tukey. It concerns a situation where there are many independent tests of significance and one is interested in rejecting the joint null hypothesis. Tukey suggested comparing the fraction of observed significances at a given �-level to the expected fraction under the joint null. In fact, he suggested standardizing the difference of the two quantities and forming a z-score; the resulting z-score tests the significance of the body of significance tests. We consider a generalization, where we maximize this z-score over a range of significance levels 0 < � � �0. We are able to show that the resulting higher criticism statistic is effective at resolving a very subtle testing problem: testing whether n normal means are all zero versus the alternative that a small fraction is nonzero. The subtlety of this “sparse normal means” testing problem can be seen from work of Ingster and Jin, who studied such problems in great detail. In their studies, they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution of the p-values either for the bulk of the tests or for the few most highly significant tests. In this range, when the amplitude of nonzero means is calibrated with the fraction of nonzero means, the likelihood ratio test for a precisely specified alternative would still succeed in separating the two hypotheses. We show that the higher criticism is successful throughout the same region of amplitude sparsity where the likelihood ratio test

Subfields: Analytical Chemistry, Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/649b8ca2acdf96b2635e19008d014e1281a34410

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Tests for High-Dimensional Covariance Matrices (2010, methodology): “…research works on inference for means of highdimensional distributions either in the context of multiple tests as in van der Laan and Bryan (2001), Donoho and Jin (2004), Fan, Hall, and Yao (2007), Kosorok and Ma (2007), and Hall and Jin (2008), or in the context of simultaneous multivariate…”
A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing (2008, methodology): “The magnitude of signals used in our simulation study is chosen according to the suggestions included in Donoho and Jin [6], so as to fulfill the condition of detectability in very sparse mixtures.”
Analyzing gene expression data in terms of gene sets: methodological issues (2007, methodology): “The competitive test is most popular: only Goeman et al. (2004, 2005), Mansmann and Meister (2005) and Tomfohr et al. (2005) present self-contained tests. Concerning the calculation of the p-value, we make a distinction between gene sampling methods and subject sampling methods. The former bases the”
Estimation and Confidence Sets for Sparse Normal Mixtures (2006, methodology): “Detailed discussion on higher criticism can be found in [5].”
Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence (2013, methodology): “According to Donoho and Jin (2004), under H0, P (a(log p)M̂0n − b(log p, η)≤ x)→ exp(−e−x), with the same normalizing sequences as those in Theorem 2.”
Genome-wide association analysis of metabolic traits in a birth cohort from a founder population (2008, methodology): “We tested the global null hypothesis of no genetic effect using the Higher Criticis”
Hierarchical testing of variable importance (2008, methodology): “As a third alternative, we consider Higher Criticism, which was introduced by Donoho and Jin (2004), based on a proposal by Tukey, and was shown to posses certain optimal properties for multiple testing when there are just very few false null hypotheses (Donoho and Jin, 2004).”
Adaptive Global Testing for Functional Linear Models (2014, background): “Such a framework (also known as the detection boundary problem) has been studied in highdimensional and nonparametric testing problems such as Ingster (1982), Donoho and Jin (2004), Ingster et al. (2010), Ingster et al. (2012), Arias-Castro et al. (2011).”
Finding and Leveraging Structure in Learning Problems (2012, background): “This should be contrasted to the unstructured normal means problem, where the bounds for localization and detection differ only in constants [65].”
Asymptotic Properties of Bayes Risk of a General Class of Shrinkage Priors in Multiple Hypothesis Testing Under Sparsity (2013, background): “See, for example, Ingster (1997), Donoho and Jin (2004), Meinshausen and Rice (2006), Cai et al. (2007) and Cai and Jin (2010), in this context.”
Optimal detection of heterogeneous and heteroscedastic mixtures (2011, methodology): “See Donoho and Jin (2004, 2008, 2009) for further discussions.”
To How Many Simultaneous Hypothesis Tests Can Normal, Student's t or Bootstrap Calibration Be Applied? (2006, background): “Examples in the latter setting have been given by Benjamini and Yekutieli (2001), Sarkar (2002, 2006), Dudoit, Shaffer, and Boldrick (2003), Donoho and Jin (2004), Efron (2004), Genovese and Wasserman (2004), Storey, Taylor, and Siegmund (2004), Lehmann and Romano (2005), and Lehmann, Romano, and…”
Detection boundary in sparse regression (2010, background): “We denote by F p ( t ) the empirical distribution of ( q i ) 1 i p : Then, the normalized uniform empirical process is deﬁned by p Arguing as in Donoho and Jin [8], we observe that t HC = sup t 1 / 2 W p ( t ).”
Proportion of non‐zero normal means: universal oracle equivalences and uniformly consistent estimators (2008, background): “The subtlety of the sparse case has been addressed in detail in Ingster (1997, 1999), Donoho and Jin (2004) and Cai et al. (2007). It is surprising that the proportion may not be estimable even when all non-zero μjs tend to ∞ uniformly.”
Detection of correlations (2011, background): “By Ery Arias-Castro1, Sébastien Bubeck and Gábor Lugosi2”
On the Benjamini-Hochberg method (2006, background): “, [5, 6, 13]), but this seems to be the first explicit link to be exhibited.”