Combining p-values via averaging

By Vladimir G Vovk, Ruodu Wang. 2012

239 citations recorded by the source corpus.

This paper proposes general methods for the problem of multiple testing of a single hypothesis, with a standard goal of combining a number of $p$-values without making any assumptions about their dependence structure. A result by Rüschendorf (1982) and, independently, Meng (1993) implies that the $p$-values can be combined by scaling up their arithmetic mean by a factor of 2, and no smaller factor is sufficient in general. A similar result by Mattner about the geometric mean replaces 2 by e. Based on more recent developments in mathematical finance, specifically, robust risk aggregation techniques, we extend these results to generalized means; in particular, we show that $K$ $p$-values can be combined by scaling up their harmonic mean by a factor of $\log K$ asymptotically as $K$ tends to infinity. This leads to a generalized version of the Bonferroni–Holm procedure. We also explore methods using weighted averages of $p$-values. Finally, we discuss the efficiency of various methods of combining $p$-values and how to choose a suitable method in light of data and prior information.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (2018, methodology): “While several less conservative aggregations methods have been proposed [18, 32, 46, 53, 55], they typically require assumptions on the dependencies among the tests.”
A unified treatment of multiple testing with prior knowledge using the p-filter (2017, background): “In other words, each group-level p-value at each layer can be formed by combining the elementary p-values within that group [17, 47].”
Admissible ways of merging p-values under arbitrary dependence (2020, unknown): “Precise p-merging functions are the main object studied by Vovk and Wang (2020a), where p-values are combined via averaging.”
False discovery rate control with e‐values (2020, background): “In addition, combination of multiple p-values relies heavily on dependence assumptions (Vovk and Wang, 2020), unlike that of e-values (Vovk and Wang, 2021).”
Predictive inference with the jackknife+ (2019, methodology): “The first part of this result, part (a), is derived from the work of Vovk and Wang [31] and Vovk et al. [30]; we give more details on this in Section 3.2 below.”