The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences

By Craig Hedge, Georgina Powell, Petroc Sumner. Behavior Research Methods, 2017

1,398 citations recorded by the source corpus.

Individual differences in cognitive paradigms are increasingly employed to relate cognition to brain structure, chemistry, and function. However, such efforts are often unfruitful, even with the most well established tasks. Here we offer an explanation for failures in the application of robust cognitive paradigms to the study of individual differences. Experimental effects become well established – and thus those tasks become popular – when between-subject variability is low. However, low between-subject variability causes low reliability for individual differences, destroying replicable correlations with other factors and potentially undermining published conclusions drawn from correlational relationships. Though these statistical issues have a long history in psychology, they are widely overlooked in cognitive psychology and neuroscience today. In three studies, we assessed test-retest reliability of seven classic tasks: Eriksen Flanker, Stroop, stop-signal, go/no-go, Posner cueing, Navon, and Spatial-Numerical Association of Response Code (SNARC). Reliabilities ranged from 0 to .82, being surprisingly low for most tasks given their common use. As we predicted, this emerged from low variance between individuals rather than high measurement variance. In other words, the very reason such tasks produce robust and easily replicable experimental effects – low between-participant variability – makes their use as correlational tools problematic. We demonstrate that taking such reliability estimates into account has the potential to qualitatively change theoretical conclusions. The implications of our findings are that well-established approaches in experimental psychology and neuropsychology may not directly translate to the study of individual differences in brain structure, chemistry, and function, and alternative metrics may be required.

Subfields: Cognitive Neuroscience, Food Science, General Decision Sciences

View Paper: https://www.semanticscholar.org/paper/9c570ea4b16fbda5029aab4cc972f2ebe4f30d2f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Food Science

In General Decision Sciences

Citations in the corpus

Reported self-control is not meaningfully associated with inhibition-related executive function: A Bayesian analysis (2018, background): “Another limitation is the previously mentioned reliability paradox (Hedge et al., 2017): Robust cognitive tasks do not produce reliable individual differences, making their use as trait-level correlational tools problematic.”
Individual Differences in Children’s Pragmatic Ability: A Review of Associations with Formal Language, Social Cognition, and Executive Functions (2018, background): “Fourth, once we have specific hypotheses about mechanisms, we need to develop tests of pragmatic and related skills that lend themselves to individual differences research (Cronbach, 1957; Hedge et al., 2017).”
Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements (2019, methodology): “Many of authors have made this same point (for a few examples, see Appelbaum et al., 2018; Cooper et al., 2017; Hedge et al., 2018; LeBel & Paunonen, 2011; and Wilkinson, 1999).”
Large-scale analysis of test–retest reliabilities of self-regulation measures (2019, background): “(33) argue, behavioral tasks designed with the subtraction logic to isolate specific cognitive processes become well-established in the literature precisely for their low between-subject variability.”