Statistical Applications in Genetics and Molecular Biology Permutation P -values Should Never Be Zero: Calculating Exact P -values When Permutations Are Randomly Drawn

By Belinda Phipson, The Walter Belinda Phipson, Eliza Hall, Gordon K. Smyth. 2011

649 citations recorded by the source corpus.

Permutation tests are amongst the most commonly used statistical tools in modern genomic research, a process by which p-values are attached to a test statistic by randomly permuting the sample or gene labels. Yet permutation p-values published in the genomic literature are often computed incorrectly, understated by about 1/m, where m is the number of permutations. The same is often true in the more general situation when Monte Carlo simulation is used to assign p-values. Although the p-value understatement is usually small in absolute terms, the implications can be serious in a multiple testing context. The understatement arises from the intuitive but mistaken idea of using permutation to estimate the tail probability of the test statistic. We argue instead that permutation should be viewed as generating an exact discrete null distribution. The relevant literature, some of which is likely to have been relatively inaccessible to the genomic community, is reviewed and summarized. A computation strategy is developed for exact p-values when permutations are randomly drawn. The strategy is valid for any number of permutations and samples. Some simple recommendations are made for the implementation of permutation tests in practice.

Subfields: Genetics, Molecular Biology, Plant Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

In Plant Science

Citations in the corpus

Kernel‐based tests for joint independence (2016, background): “Romano, J. P. (1986). A bootstrap revival of some nonparametric tests. Technical Report 254, Department of Statistics, Stanford University. Romano, J. P. (1988). A bootstrap revival of some nonparametric distance tests. Journal of the American Statistical Association 83 (403), 698–708. Romano, J. P.”
NetCoMi: network construction and comparison for microbiome data in R (2020, positive): “Since P - values of exactly zero cannot be corrected for multiple testing, NetCoMi ’s bootstrap p-values are corrected by adding pseudo counts: where b is the number of generated samples with a test statistic at least as extreme as the observed one, and m is the number of repetitions [75].”
regioneR: an R/Bioconductor package for the association analysis of genomic regions based on permutation tests (2015, methodology): “The result of performing a permutation test with regioneR is an object with the random distribution of the evaluation function, the P value (Phipson and Smyth, 2010) and the z-score of the test (Fig.”
Fast gene set enrichment analysis (2016, positive): “These two formulas follow Subramanian et al. implementa-264 tion, except of +1 terms, which are recommended by Phipson and Smyth [9].”
Comprehensive literature review and statistical considerations for microarray meta-analysis (2012, background): “According to Phipson and Smyth (131), the P -values from permutation analysis should never be zero but the ‘siggenes’ package does occasionally generate zero P -values.”
Proving Test Set Contamination in Black Box Language Models (2023, methodology): “√ By independence and the finite second moment condition, s → N (0 , σ 2 / m ) under the null by the central limit theorem and a one sided t-test provides asymptotically valid p-values with Preprint P ( p α ) → α uniformly as m → ∞ (Lehmann & Romano, 2005, Theorem 11.4.5).”
Camera: a competitive gene set test accounting for inter-gene correlation (2012, methodology): “For the one-sided directional test S 1⁄4 c versus the alternative S c, the P-value is the proportion of the gene sets, combining the random sets S* with the original S (33), for which the mean statistic z S zS.”
Fast and efficient QTL mapper for thousands of molecular phenotypes (2015, positive): “More formally, if r correlations in the null distribution are found to be stronger than the observed, significance of the QTL candidate is assessed by calculating the following empirical adjusted P-value pd of association (Phipson and Smyth, 2010):”
Participatory provenance as representational auditing for AI-mediated public consultation (2026, unknown): “For the random and style-matched baselines, one-sided empirical p -values with the +1 correction of Phipson and Smyth [Phipson and Smyth, 2010] were the primary test statistics, testing whether the actual summary’s mean coverage (lower tail) and W 2 (upper tail) are extreme relative to the baseline…”