Robustness of multiple testing procedures against dependence

By S. Clarke, Peter Hall. 2009

90 citations recorded by the source corpus.

An important aspect of multiple hypothesis testing is controlling the significance level, or the level of Type I error. When the test statistics are not independent it can be particularly challenging to deal with this problem, without resorting to very conservative procedures. In this paper we show that, in the context of contemporary multiple testing problems, where the number of tests is often very large, the difficulties caused by dependence are less serious than in classical cases. This is particularly true when the null distributions of test statistics are relatively light-tailed, for example, when they can be based on Normal or Student's t approximations. There, if the test statistics can fairly be viewed as being generated by a linear process, an analysis founded on the incorrect assumption of independence is asymptotically correct as the number of hypotheses diverges. In particular, the point process representing the null distribution of the indices at which statistically significant test results occur is approximately Poisson, just as in the case of independence. The Poisson process also has the same mean as in the independence case, and of course exhibits no clustering of false discoveries. However, this result can fail if the null distributions are particularly heavy-tailed. There clusters of statistically significant results can occur, even when the null hypothesis is correct. We give an intuitive explanation for these disparate properties in light- and heavy-tailed cases, and provide rigorous theory underpinning the intuition.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/0b9930266112c8b0684343f741cf83cf2b9a1da2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Journal of the American Statistical Association Estimating False Discovery Proportion under Arbitrary Covariance Dependence Estimating False Discovery Proportion under Arbitrary Covariance Dependence (unknown): “Although Benjamini and Yekutieli (2001) and Clarke and Hall (2009) argued that when the null distribution of test statistics satisfies some conditions, the dependence case in the multiple testing is asymptotically the same as the independence case, multiple testing under general dependence…”
Mass univariate analysis of event-related brain potentials/fields I: A critical tutorial review (2011, background): “Theoretical results show that when data come from relatively light-tailed distributions (e.g., a normal distribution), FDR control performs as if the tests were independent as the number of tests in a family increases (Clarke & Hall, 2009).”
Large-scale inference (2010, background): “Other good references for large-scale correlation effects include Dudoit, van der Laan and Pollard (2004), Owen (2005), Qiu, Klebanov and Yakovlev (2005b), Qiu, Brooks, Klebanov and Yakovlev (2005a), Desai, Deller and McCormick (2009), and Clarke and Hall (2009). Schwartzman and Lin (2009) use Mehle”
Asymptotic optimality of the Westfall--Young permutation procedure for multiple testing under dependence (2011, methodology): “Furthermore, Clarke and Hall [7] discuss the effect of dependence and clustering when using “wrong” methods based on independence assumptions for controlling the (generalized) FWER and FDR.”
A Selective Overview of Variable Selection in High Dimensional Feature Space (2009, background): “See also Efron et al. (2001), Abramovich et al. (2006), Donoho and Jin (2006), and Clarke and Hall (2009).”