To How Many Simultaneous Hypothesis Tests Can Normal, Student's t or Bootstrap Calibration Be Applied?

By Jianqing Fan, Peter Hall, Qiwei Yao. 2006

88 citations recorded by the source corpus.

In the analysis of microarray data, and in some other contemporary statistical problems, it is not uncommon to apply hypothesis tests in a highly simultaneous way. The number, N say, of tests used can be much larger than the sample sizes, n, to which the tests are applied, yet we wish to calibrate the tests so that the overall level of the simultaneous test is accurate. Often the sampling distribution is quite different for each test, so there may not be an opportunity to combine data across samples. In this setting, how large can N be, as a function of n, before level accuracy becomes poor? Here we answer this question in cases where the statistic under test is of Student's t type. We show that if either the normal or Student t distribution is used for calibration, then the level of the simultaneous test is accurate provided that log N increases at a strictly slower rate than n 1/3 as n diverges. On the other hand, if bootstrap methods are used for calibration, then we may choose log N almost as large as n 1/2 and still achieve asymptotic-level accuracy. The implications of these results are explored both theoretically and numerically.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/918f6216cf0c7e95825737fdef804cba610d0785

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

STEPUP PROCEDURES CONTROLLING GENERALIZED FWER AND GENERALIZED FDR (2007, background): “expected proportion of falsely rejected null hypotheses and which was introduced by Benjamini and Hochberg [1], is the first of these that has received considerable attention [2, 3, 4 , 5, 6, 12, 15, 16, 17, 21, 22].”
RAPTT: An Exact Two-Sample Test in High Dimensions Using Random Projections (2014, background): “…Kuelbs and Vidyashankar (2010); Clemencon et al. (2009); Jacob et al. (2010); Lu et al. (2005) studied inference for means in high dimensions; Fan et al. (2007) worked on simultaneous testing of means based on marginal tests in high dimensions; Benjamini and Hochberg (1995) worked on…”
Two Sample Tests for High Dimensional Covariance Matrices (2012, methodology): “…for means of high-dimensional distributions either in the context of multiple testing, as in van der Laan and Bryan (2001), Donoho and Jin (2004), Fan, Hall and Yao (2007) and Hall and Jin (2008), or in This is an electronic reprint of the original article published by the Institute of…”
Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors (2012, background): “The question on how large p can be was studied in [22] but from a conservative perspective.”
Tests for High-Dimensional Covariance Matrices (2010, methodology): “…for means of highdimensional distributions either in the context of multiple tests as in van der Laan and Bryan (2001), Donoho and Jin (2004), Fan, Hall, and Yao (2007), Kosorok and Ma (2007), and Hall and Jin (2008), or in the context of simultaneous multivariate testing as in Bai and…”