Control of generalized error rates in multiple testing

By Joseph P. Romano, Michael Wolf. 2007

179 citations recorded by the source corpus.

Consider the problem of testing s hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If s is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of k or more false rejections, which is called the k-FWER. We derive both single-step and step-down procedures that control the k-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289-300] controls E(FDP). Here, the goal is to construct methods which satisfy, for a given y and a, P{FDP >y} ≤ a, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138-1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3-9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/1c09afaa4bdc00a66643c3d16504dec85474cde0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Stepup procedures for control of generalizations of the familywise error rate (2006, methodology): “Our methods are designed to hold under no dependence assumptions among the p-values, but do not attempt to estimate the dependence structure (as in van der Laan, Dudoit and Pollard [16] or Romano and Wolf [13]).”
Hypothesis Testing in Econometrics (2010, background): “be quite anti-conservative; see Romano and Wolf (2007) .”
Multiple Testing for Exploratory Research (2011, unknown): “A variant of familywise error, k-FWER, has been formulated that controls the probability of making at least k ≥ 1 false rejections (Romano and Wolf, 2007).”
Statistical arbitrage pairs trading strategies: Review and outlook (2017, unknown): “After rigorously controlling for data snooping bias following procedures of Romano and Wolf (2007) as well as Romano et al. (2010), they find that the daily returns of their top strategies amount to 0.07 to 0.55 percent.”
Distribution-free Multiple Testing (2016, background): “Except for those cited above, all the other ones we know of are based on resampling (Ge et al., 2003; Romano and Wolf, 2007; Westfall and Young, 1993; Yekutieli and Benjamini, 1999).”
Familywise error rate control via knockoffs (2015, methodology): “(a) the generic step-down procedure of Romano and Wolf (2007) applied to the least-squares p-values”
Control of the false discovery rate under dependence using the bootstrap and subsampling (2008, background): “Methods that incorporate such information and are thus better able to detect false null hypotheses are described in van der Laan et al. (2004), Romano and Wolf (2007), and Romano et al. (2008).”
Balanced Control of Generalized Error Rates (2010, methodology): “Note that, in comparison, the step-down methods developed in [31] use a common critical value at each stage of the algorithm, which does not depend on i.”