Performing High-Powered Studies Efficiently with Sequential Analyses

By Daniël Lakens. 2014

344 citations recorded by the source corpus.

Running studies with high statistical power, while effect size estimates in psychology are often inaccurate, leads to a practical challenge when designing an experiment. This challenge can be addressed by performing sequential analyses while the data collection is still in progress. At an interim analysis, data collection can be stopped whenever the results are convincing enough to conclude an effect is present, more data can be collected, or the study can be terminated whenever it is extremely unlikely the predicted effect will be observed if data collection would be continued. Such interim analyses can be performed while controlling the Type 1 error rate. Sequential analyses can greatly improve the efficiency with which data is collected. Additional flexibility is provided by adaptive designs where sample sizes are increased based on the observed effect size. The need for pre-registration, ways to prevent experimenter bias, and a comparison between Bayesian approaches and NHST are discussed. Sequential analyses, which are widely used in large scale medical trials, provide an efficient way to perform high-powered informative experiments. I hope this introduction will provide a practical primer that allows researchers to incorporate sequential analyses in their research.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/0730428e74ee2da93ff2e57d33c3a10b6d3e230a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Rejecting or Accepting Parameter Values in Bayesian Estimation (2018, unknown): “…of equivalence, equivalence margin, margin of noninferiority, smallest effect size of interest, and good-enough belt (e.g., Carlin & Louis, 2009; Freedman, Lowe, & Macaskill, 1984; Hobbs & Carlin, 2008; Lakens, 2014, 2017; Serlin & Lapsley, 1985, 1993; Spiegelhalter, Freedman, & Parmar, 1994).”
When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias (2018, background): “For a frequentist introduction of sequential analysis, see Lakens (2014), for a Bayesian introduction, see Schönbrodt, Wagenmakers, Zehetleitner, and Perugini (2017), for the mathematical background, see Wald (1945), and Siegmund (2013).”
Sample Size Justification (2021, unknown): “Sequential designs are especially useful when there is considerableuncertaintyabouttheeffectsize, orwhen it is plausible that the true effect size is larger than the smallest effect size of interest the study is designed to detect (Lakens, 2014).”
Pre-registration in social psychology—A discussion and suggested template (2016, methodology): “Manuscripts then often spend a long half-life bouncing from journal to journal, until they either add new and better evidence, or they find a combination of editor and reviewers willing to overlook the flaws (Nosek & Bar-Anan, 2012).”
The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective (2017, unknown): “…“equivalence interval,” “indifference zone,” “smallest effect size of interest,” and “good-enough belt” (e.g. Carlin & Louis, 2009; Freedman, Lowe, & Macaskill, 1984; Hobbs & Carlin, 2008; Lakens, 2014; Schuirmann, 1987; Serlin & Lapsley, 1985, 1993; Spiegelhalter, Freedman, & Parmar, 1994).”
Equivalence Tests (2017, unknown): “Therefore, setting equivalence boundaries to your SESOI in an equivalence test allows you to reject effect sizes larger than you find worthwhile to examine, given available resources.”
Using Anchor-Based Methods to Determine the Smallest Effect Size of Interest (2019, methodology): “The more general description of 'smallest effect size of interest' refers to the smallest effect size that is predicted by theoretical models, considered relevant in daily life, or that is feasible to study empirically (Lakens, 2014).”
Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods (2019, background): “0030 Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44 (7), 701–710. Light, R. J. & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Harvard University Press. Macaskill, P., Walter, S. D., & Irwig,”