Multiple Testing for Exploratory Research

By Jelle J. Goeman, Aldo Solari. 2011

242 citations recorded by the source corpus.

Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing.

Subfields: Management Science and Operations Research, Sociology and Political Science, Software

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Science and Operations Research

In Sociology and Political Science

In Software

Citations in the corpus

Invariant Causal Prediction for Nonlinear Models (2017, background): “The results are to some extent comparable to similar issues arising in multiple testing (Goeman and Solari, 2011).”
Permutation-based simultaneous confidence bounds for the false discovery proportion (2018, methodology): “Most of these methods (including those in the present paper) are special cases or shortcuts for the general methods in Genovese and Wasserman (2006) and Goeman and Solari (2011).”
Only closed testing procedures are admissible for controlling false discovery proportions (2019, methodology): “As proposed by Goeman and Solari (2011), procedures that fulfil (1) or (2) allow a different, flexible way of using multiple testing methods.”
Multiple hypothesis testing in genomics (2014, methodology): “The potential for FDP estimation, as opposed to FDR or FWER control, is therefore that it allows researchers to approach multiple testing in a different way that reverses the traditional roles of the user and the multiple testing procedure [12].”
Predicting the effect of interventions using invariance principles for nonlinear models (2016, background): “We can, however, extract more information via defining sets comparable to a similar issue arising in multiple testing [4].”
Admissible ways of merging p-values under arbitrary dependence (2020, unknown): “Our p-merging functions can be directly applied to any procedures for multiple hypothesis testing, such as those of Genovese and Wasserman (2004) and Goeman and Solari (2011); see Section 10 for simulation studies. In addition to the grid harmonic p-merging function H∗ K , strictly dominating the me”
E-values: Calibration, combination and applications (2019, unknown): “As we explain in Appendix A (Section A.3), our algorithms just spell out the application of the closure principle [Marcus et al., 1976; Goeman and Solari, 2011], but our exposition in this section will be self-contained.”
Simultaneous control of all false discovery proportions in large-scale multiple hypothesis testing (2016, methodology): “Goeman and Solari (2011) show how to calculate upper confidence bounds tα(S) for τ(S) from the closed testing procedure of Marcus et al. (1976). In the general closed testing procedure, the collection of hypotheses is augmented with all possible intersection hypotheses HI = ⋂”