Psychological Science Needs a Standard Practice of Reporting the Reliability of Cognitive-Behavioral Measurements
By S. Parsons, A. Kruijt, E. Fox. Advances in Methods and Practices in Psychological Science, 2019
377 citations recorded by the source corpus.
Psychological science relies on behavioral measures to assess cognitive processing; however, the field has not yet developed a tradition of routinely examining the reliability of these behavioral measures. Reliable measures are essential to draw robust inferences from statistical analyses, and subpar reliability has severe implications for measures’ validity and interpretation. Without examining and reporting the reliability of measurements used in an analysis, it is nearly impossible to ascertain whether results are robust or have arisen largely from measurement error. In this article, we propose that researchers adopt a standard practice of estimating and reporting the reliability of behavioral assessments of cognitive processing. We illustrate the need for this practice using an example from experimental psychopathology, the dot-probe task, although we argue that reporting reliability is relevant across fields (e.g., social cognition and cognitive psychology). We explore several implications of low measurement reliability and the detrimental impact that failure to assess measurement reliability has on interpretability and comparison of results and therefore research quality. We argue that researchers in the field of cognition need to report measurement reliability as routine practice so that more reliable assessment tools can be developed. To provide some guidance on estimating and reporting reliability, we describe the use of bootstrapped split-half estimation and intraclass correlation coefficients to estimate internal consistency and test-retest reliability, respectively. For future researchers to build upon current results, it is imperative that all researchers provide psychometric information sufficient for estimating the accuracy of inferences and informing further development of cognitive-behavioral assessments.
Subfields: Applied Psychology, Cognitive Neuroscience, Experimental and Cognitive Psychology
View Paper: https://www.semanticscholar.org/paper/cbfb887a6b231b2a1d58b451e5c3ef479f17cf9a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Applied Psychology
In Cognitive Neuroscience
In Experimental and Cognitive Psychology
Citations in the corpus
- Sample Size Justification (2021, neutral): “Finally, it is important to know the reliability of your measure (Parsons, Kruijt, & Fox, 2019), especially when relying on an effect size from a published study that used a measure with different reliability, or when the same measure is used in different populations, in which case it is possible…”
- Psychological measurement and the replication crisis: Four sacred cows. (2020, methodology): “Parsons et al. (2019) recently furnished readers with a set of user-friendly guidelines, along with a tutorial and R code, for calculating the internal consistencies of laboratory-based indices.”