Statistically Controlling for Confounding Constructs Is Harder than You Think

By J. Westfall, Tal Yarkoni. PLoS ONE, 2016

440 citations recorded by the source corpus.

Social scientists often seek to demonstrate that a construct has incremental validity over and above other related constructs. However, these claims are typically supported by measurement-level models that fail to consider the effects of measurement (un)reliability. We use intuitive examples, Monte Carlo simulations, and a novel analytical framework to demonstrate that common strategies for establishing incremental construct validity using multiple regression analysis exhibit extremely high Type I error rates under parameter regimes common in many psychological domains. Counterintuitively, we find that error rates are highest—in some cases approaching 100%—when sample sizes are large and reliability is moderate. Our findings suggest that a potentially large proportion of incremental validity claims made in the literature are spurious. We present a web application (http://jakewestfall.org/ivy/) that readers can use to explore the statistical properties of these and other incremental validity arguments. We conclude by reviewing SEM-based statistical approaches that appropriately control the Type I error rate when attempting to establish incremental validity.

Subfields: Experimental and Cognitive Psychology, Social Psychology, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/0a1e930a87c0f634549c6d0ed64fb6b7f2d07ca2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Social Psychology

In Sociology and Political Science

Citations in the corpus

Meaningful Mediation Analysis: Plausible Causal Inference and Informative Communication (2017, methodology): “Importantly, structural equation modeling can also account for measurement error when mediator and/or outcome are measured with single items, if estimates of their reliability are available from previous studies, meta-analyses, or other sources (Anderson and Gerbing 1988; Westfall and Yarkoni 2016).”
Rapid Systematic Review: The Impact of Social Isolation and Loneliness on the Mental Health of Children and Adolescents in the Context of COVID-19 (2020, background): “In cross-sectional studies, likely confounders (eg, adversity, socioeconomic status [SES]) were rarely controlled for, meaning that the association between loneliness and mental health outcomes in these studies is very likely to be inflated.(16) Four longitudinal studies used multi-informant approac”
Using independent covariates in experimental designs: Quantifying the trade-off between power boost and Type I error inflation (2017, background): “We focus here on the first context; a host of other issues are relevant for the second context and have been discussed in detail elsewhere (e.g., Westfall & Yarkoni, 2016; Yzerbyt, Muller, & Judd, 2004).”
Federated Learning in Distributed Medical Databases: Meta-Analysis of Large-Scale Subcortical Brain Data (2018, background): “Interestingly, the issue of “leaking” spurious variability of confounders after correction has been already reported in a number of multi-centric studies, and is matter of ongoing research [14, 15].”
Statistical Challenges in "Big Data" Human Neuroimaging. (2018, positive): “…is incomplete and increasing N just increases sensitivity to the un-adjusted variance that ‘leaks’ through ; addressing this requires measures of confound uncertainty and a more complex modelling approach that propagates this uncertainty through the modelling process (Westfall and Yarkoni 2014).”
Thinking Clearly About Correlations and Causation: Graphical Causal Models for Observational Data (2017, background): “Westfall and Yarkoni (2016) assessed how the false positive rate for an effect is affected by measurement error of covariates that are being controlled for.”
Objections to the HEXACO Model of Personality Structure—and why those Objections Fail (2020, background): “As discussed by Schmidt, Hunter, and Caplan (1981) and by Westfall and Yarkoni (2016), the incremental validity provided by one scale over another can sometimes occur simply because the former helps to make up for the imperfect reliability of the latter.”
Choosing prediction over explanation in psychology: Lessons from machine learning (2017, background): “For example, we and others have demonstrated that under typical measurement conditions, statistical significance patterns observed in fitted regression models are often very unstable and can produce highly misleading conclusions (Shear & Zumbo, 2013; Westfall & Yarkoni, 2016).”