Estimating Population Mean Power Under Conditions of Heterogeneity and Selection for Significance

By J. Brunner, U. Schimmack. 2020

51 citations recorded by the source corpus.

In scientific fields that use significance tests, statistical power is important for successful replications of significant results because it is the long-run success rate in a series of exact replication studies. For any population of published results, there is a population of power values of the statistical tests on which conclusions are based. We give exact theoretical results showing how selection for significance affects the distribution of statistical power in a heterogeneous population of significance tests. In a set of large-scale simulation studies, we compare four methods for estimating population mean power of a set of studies selected for significance (a maximum likelihood model, extensions of p-curve and p-uniform, & z-curve). The p-uniform and p-curve methods performed well with a fixed effects size and varying sample sizes. However, when there was substantial variability in effect sizes as well as sample sizes, both methods systematically overestimate mean power. When the assumptions of the maximum likelihood were satisfied, it produced the most accurate estimates for heterogeneity in effect sizes, but z-curve produced more accurate estimates when the assumptions of the maximum likelihood model were not met. We recommend the use of zcurve to estimate the typical power of significant results, which has implications for the replicability of significant results in psychology journals.

Subfields: Experimental and Cognitive Psychology, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c8ccf44582872acbde1f98f0739eafd56b56a5b9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Statistics and Probability

Citations in the corpus

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis (2021, methodology): “We bypassed these problems by computing the z-score for each statistical result’s p-value, which is also done before analyzing data in Stouffer’s method in meta-analysis and z-curve (Brunner & Schimmack, 2018).”