When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias

By Casper J. Albers, Daniël Lakens. 2018

303 citations recorded by the source corpus.

Abstract When designing a study, the planned sample size is often based on power analyses. One way to choose an effect size for power analyses is by relying on pilot data. A-priori power analyses are only accurate when the effect size estimate is accurate. In this paper we highlight two sources of bias when performing a-priori power analyses for between-subject designs based on pilot data. First, we examine how the choice of the effect size index (η2, ω2 and e2) affects the sample size and power of the main study. Based on our observations, we recommend against the use of η2 in a-priori power analyses. Second, we examine how the maximum sample size researchers are willing to collect in a main study (e.g. due to time or financial constraints) leads to overestimated effect size estimates in the studies that are performed. Determining the required sample size exclusively based on the effect size estimates from pilot data, and following up on pilot studies only when the sample size estimate for the main study is considered feasible, creates what we term follow-up bias. We explain how follow-up bias leads to underpowered main studies. Our simulations show that designing main studies based on effect sizes estimated from small pilot studies does not yield desired levels of power due to accuracy bias and follow-up bias, even when publication bias is not an issue. We urge researchers to consider alternative approaches to determining the sample size of their studies, and discuss several options.

Subfields: Economics and Econometrics, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/21b19354d9798724550cff5db0968746563255ef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Sample Size Justification (2021, unknown): “…effect size in the pilot study that, when entered into a power analysis, yields a sample size that is feasible to collect for the follow-up study, these effect size estimates will be upwardly biased, and power in the follow-up study will be systematically lower than desired (Albers & Lakens, 2018).”
Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R (2020, background): “One approach is choosing the smallest effect size of interest (SESOI) to run a power analysis — making it possible to design studies which are worthwhile to run, as they have a predetermined power to detect an effect that is of interest (Albers & Lakens, 2018).”
A Simple Method for Removing Bias From a Popular Measure of Standardized Effect Size: Adjusted Partial Eta Squared (2019, background): “It should be noted, however, that removing bias from η p 2 has the consequence of increasing its variability (see Albers & Lakens, 2018, for a clear demonstration).”
Using Anchor-Based Methods to Determine the Smallest Effect Size of Interest (2019, background): “If a SESOI can be determined, it is possible to choose a sample size that provides high power for the smallest effect size that is deemed meaningful (Albers & Lakens, 2018).”