Lack of group-to-individual generalizability is a threat to human subjects research

By A. Fisher, John D. Medaglia, Bertus F. Jeronimus. Proceedings of the National Academy of Sciences of the United States of America, 2018

852 citations recorded by the source corpus.

Significance The current study quantified the degree to which group data are able to describe individual participants. We utilized intensive repeated-measures data—data that have been collected many times, across many individuals—to compare the distributions of bivariate correlations calculated within subjects vs. those calculated between subjects. Because the vast majority of social and medical science research aggregates across subjects, we aimed to assess how closely such aggregations reflect their constituent individuals. We provide evidence that conclusions drawn from aggregated data may be worryingly imprecise. Specifically, the variance in individuals is up to four times larger than in groups. These data call for a focus on idiography and open science that may substantially alter best-practice guidelines in the medical and behavioral sciences. Only for ergodic processes will inferences based on group-level data generalize to individual experience or behavior. Because human social and psychological processes typically have an individually variable and time-varying nature, they are unlikely to be ergodic. In this paper, six studies with a repeated-measure design were used for symmetric comparisons of interindividual and intraindividual variation. Our results delineate the potential scope and impact of nonergodic data in human subjects research. Analyses across six samples (with 87–94 participants and an equal number of assessments per participant) showed some degree of agreement in central tendency estimates (mean) between groups and individuals across constructs and data collection paradigms. However, the variance around the expected value was two to four times larger within individuals than within groups. This suggests that literatures in social and medical sciences may overestimate the accuracy of aggregated statistical estimates. This observation could have serious consequences for how we understand the consistency between group and individual correlations, and the generalizability of conclusions between domains. Researchers should explicitly test for equivalence of processes at the individual and group level across the social and medical sciences.

Subfields: Applied Psychology, Experimental and Cognitive Psychology, Social Psychology

View Paper: https://www.semanticscholar.org/paper/d441ac8a4da15b1ebd9321f33c914647d99b780d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

ARTIFICIAL INTELLIGENCE AND PERSONALIZED MEDICINE (2019, background): “[68] This potential lack of ‘ergodicity’ could result in models that are not useful for making individual treatment decisions.”
Descriptive, Predictive and Explanatory Personality Research: Different Goals, Different Approaches, but a Shared Need to Move beyond the Big Few Traits (2020, background): “But the extent to which this is appropriate needs to be studied not presumed (e.g., Molenaar & Campbell, 2009; Fisher et al., 2018).”
Strengthening of alpha synchronization is a neural correlate of cognitive transfer (2024, background): “Second, individual behavioral responses to cognitive training differ to the group-averaged response (Ericson and Klingberg 2023), which implies that the group averaged neural dynamics might not generalize well either (Fisher et al. 2018).”