Generalizations of the familywise error rate

By Erich Leo Lehmann, Joseph P. Romano. 2005

437 citations recorded by the source corpus.

Consider the problem of simultaneously testing null hypotheses H 1, … H s . The usual approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, particularly if s- is large, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which we call the k-FWER. We derive both single-step and stepdown procedures that control the k-FWER, without making any assumptions concerning the dependence structure of the p-values of the individual tests. In particular, we derive a stepdown procedure that is quite simple to apply, and prove that it cannot be improved without violation of control of the k-FWER. We also consider the false discovery proportion (FDP) defined by the number of false rejections divided by the total number of rejections (defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289–300] controls E(FDP). Here, we construct methods such that, for any γ and α, P{FDP > γ} ≤ α. Two stepdown methods are proposed. The first holds under mild conditions on the dependence structure of p-values, while the second is more conservative but holds without any dependence assumptions.

Subfields: Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/3df8caeca6fc1d7aab6f4daf42c2ea6e636dfb6d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Exact calculations for false discovery proportion with application to least favorable configurations (2010, background): “Alternatively, some authors are interested in procedures R controlling the FDP, that is, satisfying P[FDP(R) ≤ α] ≥ 1 − γ ; see, for example, [19, 22].”
Two simple sufficient conditions for FDR control (2008, methodology): “This intuition is in accordance with a case of equality in Hommel’s inequality established by Lehmann and Romano (2005, Lemma 3.1 (ii) ).”
Exceedance Control of the False Discovery Proportion (2006, methodology): “Multiple testing procedures to control the false discovery rate (FDR), first introduced by Benjamini and Hochberg (1995), have recently received much attention (see Benjamini and Yekutieli 2001; Efron, Tibshirani, and Storey 2001; Finner, and Roters 2002; Genovese and Wasserman 2002, 2004; Korn, Tro”
Hypothesis Testing in Econometrics (2010, methodology): “Both the Lehmann and Romano (2005a) method to control the FDP and the”
Control of the mean number of false discoveries, Bonferroni and stability of multiple testing (2007, methodology): “The main virtue of Bonf γ is its higher stability in terms of the variance of the total number of discoveries, a property that becomes increasingly important with the strength of correlation in the data [Qiu, Klebanov and Yakovlev (2005), Qiu et al. (2006)].”
Balanced Control of Generalized Error Rates (2010, background): “Hommel and Hoffman (1988) and Lehmann and Romano (2005a) , which are based on individual”
Generalizing Simes' test and Hochberg's stepup procedure (2008, methodology): “p-values, we modify the Bonferroni type single-step procedure in [12] and then develop its Holm type stepdown improvement.”
Stepup procedures for control of generalizations of the familywise error rate (2006, methodology): “Lehmann and Romano [10] provide two results that show that the stepdown procedure with critical values ααi with this choice of αi satisfies (1).”
Hidden Multiplicity in Multiway ANOVA: Prevalence, Consequences, and Remedies (2014, positive): “…such, in an explorative setting, all hypotheses implied by the design are considered and tested jointly, rendering this collection of hypotheses a family, where “... the term ‘family’ refers to the collection of hypotheses [...] that is being considered for joint testing” (Lehmann & Romano, 2005).”
Large-scale inference (2010, background): “Lehmann and Romano (2005a) develop an algorithm that controls the probability that Fdp exceeds a certain threshold, rather than E{Fdp}.”
Covariate powered cross‐weighted multiple testing (2017, methodology): “ine, we used the fact that for i2H 0 it holds that P iis super-uniform. Also, by Lemma 1, P iis independent of W i. Remark 1. The IHW-Bonferroni procedure when applied at level kcontrols the k-FWER (Lehmann and Romano, 2005b), i.e., Pr[V k]. Remark 2. While we have omitted this from the notation in ”
Only closed testing procedures are admissible for controlling false discovery proportions (2019, methodology): “A generalization, k-FWER (Finos and Farcomeni, 2011; Guo et al., 2010; Hommel and Hoffmann, 1988; Lehmann and Romano, 2005; Romano and Shaikh, 2006; Sarkar, 2007), makes sure that, for all θ ∈ Θ,”
On stepdown control of the false discovery proportion (2006, methodology): “Lehmann and Romano [10] also considered control of the false discovery proportion (FDP ), defined as the total number of false rejections divided by the total number of rejections (and equal to 0 if there are no rejections).”