Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

By J. Flake, Eiko I. Fried. 2020

635 citations recorded by the source corpus.

In this article, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons, including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical-conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, and pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.

Subfields: Experimental and Cognitive Psychology, Management Science and Operations Research, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/923e08564400d1f19bdb54de9afa6f5360aa56fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Experimental and Cognitive Psychology

In Management Science and Operations Research

In Statistics and Probability

Citations in the corpus

Use Omega Rather than Cronbach’s Alpha for Estimating Reliability. But… (2020, methodology): “…the subsets option in the OMEGA macro can assist in reducing the number of items on ameasurement instrument being developed or already in existence, it is important to remember that good measurement is about far more than establishing high reliability (see Flake & Fried, 2019; Fried & Flake, 2018).”
The Accuracy and Validity of Self-Reported Social Media Use Measures Among Adolescents (2021, background): “These adult studies found that adults tend to overestimate the time spent on their phones and the internet, and they yielded correlations ranging from r ¼ .23 (Araujo et al., 2017) to r ¼ .79 (Funch et al., 1996) between retrospective surveys and digital trace methods.”
Psychological measurement and the replication crisis: Four sacred cows. (2020, background): “As Flake and Fried (2019) observed, the field of psychological science at large has been characterized by what can be described as a measurement schmeasurement attitude (see also Hughes, 2018).”
A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use (2021, methodology): “confident that the measures we use accurately reflect the behaviour that they are designed to assess 5,20 .”