Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review.

By Herman Aguinis, James C. Beaty, Robert J. Boik, C. Pierce. Journal of Applied Psychology, 2005

1,159 citations recorded by the source corpus.

The authors conducted a 30-year review (1969-1998) of the size of moderating effects of categorical variables as assessed using multiple regression. The median observed effect size (f(2)) is only .002, but 72% of the moderator tests reviewed had power of .80 or greater to detect a targeted effect conventionally defined as small. Results suggest the need to minimize the influence of artifacts that produce a downward bias in the observed effect size and put into question the use of conventional definitions of moderating effect sizes. As long as an effect has a meaningful impact, the authors advise researchers to conduct a power analysis and plan future research designs on the basis of smaller and more realistic targeted effect sizes.

Subfields: Applied Psychology, Computer Networks and Communications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/90ec650a8c42ee61b7aad6edfe62f19941dfed39

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

The Family's Financial Support as a “Poisoned Gift”: A Family Embeddedness Perspective on Entrepreneurial Intentions (2017, background): “Besides, effect sizes that may appear rather small in absolute terms can still have substantial practical and theoretical value (Aguinis et al. 2005).”
Understanding “It Depends” in Organizational Research (2017, background): “Of course, significance tests are predicated on having sufficient power to adequately test one’s hypotheses, and the power to detect significant interactions is notoriously weak (Aguinis et al., 2005).”
CROSS-NATIONAL DIFFERENCES IN RELATIONSHIPS OF WORK DEMANDS, JOB SATISFACTION, AND TURNOVER INTENTIONS WITH WORK-FAMILY CONFLICT (2007, background): “However, they were all above the median effect size of .002 reported by Aguinis, Beaty, Boik, and Pierce (2005) in a meta-analysis of published studies reporting moderator tests.”
Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact (2011, background): “Given documented difficulties in obtaining information from authors that is not available in the published articles (Aguinis et al., 2005), we coded as much information as was available in the articles.”
ESTIMATING MODERATING EFFECTS IN PLS-SEM AND PLSc-SEM: INTERACTION TERM GENERATION*DATA TREATMENT (2018, background): “For example, Aguinis et al.’s (2005) review of all studies published from 1969 to 1998 in the Journal of Applied Psychology, Personnel Psychology, and Academy of Management Journal reports a mean moderating effect size of .”
Best-practice recommendations for estimating interaction effects using moderated multiple regression (2010, methodology): “For example, Aguinis et al. (2005) reviewed all issues of Journal of Applied Psychology (JAP), Personnel Psychology (PPsych), and Academy of Management Journal (AMJ) from 1969 to 1998 and identified all articles reporting a categorical moderator test using MMR.”
Entrepreneurs’ human capital and growth aspirations: the moderating role of regional entrepreneurial culture (2019, methodology): “In that sense, a multilevel random effects specification is more accurate than the multivariate methods (e.g., moderated multiple regressions) normally used in the management literature to estimate interaction effects (Aguinis et al. 2005).”