Large-scale analysis of test–retest reliabilities of self-regulation measures

By A. Z. Enkavi, Ian W. Eisenberg, Patrick G. Bissett, Gina L. Mazza, David P. MacKinnon, Lisa A. Marsch, R. Poldrack. Proceedings of the National Academy of Sciences of the United States of America, 2019

403 citations recorded by the source corpus.

Significance Self-regulation is a psychological construct that is characterized using a broad set of measures and is thought to be related to a number of real-world outcomes. However, the test–retest reliability of many of these measures is unclear. This paper reviews the literature on the test–retest reliability of self-regulation measures and characterizes long-term test–retest reliability in a large sample of individuals completing an extensive battery. The results show that while self-report measures have generally high test–retest reliability, behavioral task measures have substantially lower test–retest reliability, raising questions about their ability to serve as trait-like measures of individual differences. The ability to regulate behavior in service of long-term goals is a widely studied psychological construct known as self-regulation. This wide interest is in part due to the putative relations between self-regulation and a range of real-world behaviors. Self-regulation is generally viewed as a trait, and individual differences are quantified using a diverse set of measures, including self-report surveys and behavioral tasks. Accurate characterization of individual differences requires measurement reliability, a property frequently characterized in self-report surveys, but rarely assessed in behavioral tasks. We remedy this gap by (i) providing a comprehensive literature review on an extensive set of self-regulation measures and (ii) empirically evaluating test–retest reliability of this battery in a new sample. We find that dependent variables (DVs) from self-report surveys of self-regulation have high test–retest reliability, while DVs derived from behavioral tasks do not. This holds both in the literature and in our sample, although the test–retest reliability estimates in the literature are highly variable. We confirm that this is due to differences in between-subject variability. We also compare different types of task DVs (e.g., model parameters vs. raw response times) in their suitability as individual difference DVs, finding that certain model parameters are as stable as raw DVs. Our results provide greater psychometric footing for the study of self-regulation and provide guidance for future studies of individual differences in this domain.

Subfields: Applied Psychology, Experimental and Cognitive Psychology, Social Psychology

View Paper: https://www.semanticscholar.org/paper/4007b6d7b66d74aeea65af7fb2e6dc43509baea5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Experimental and Cognitive Psychology

In Social Psychology

Citations in the corpus

Self-regulation in childhood as a predictor of future outcomes: A meta-analytic review. (2020, background): “It is interesting to note that while correlations between selfregulation measures tend to be low (Duckworth & Kern, 2011; Enkavi et al., 2019) they tend to predict outcomes to a similar extent.”
Taking Skills Seriously: Toward an Integrative Model and Agenda for Social, Emotional, and Behavioral Skills (2020, background): “For example, performance measures designed to measure a particular capacity often have relatively low retest reliability and may fail to converge with one another (Eisenberg et al., 2019; Enkavi et al., 2019).”