An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results

By M. Chmielewski, Sarah C. Kucker. 2019

1,004 citations recorded by the source corpus.

Amazon’s Mechanical Turk (MTurk) is arguably one of the most important research tools of the past decade. The ability to rapidly collect large amounts of high-quality human subjects data has advanced multiple fields, including personality and social psychology. Beginning in summer 2018, concerns arose regarding MTurk data quality leading to questions about the utility of MTurk for psychological research. We present empirical evidence of a substantial decrease in data quality using a four-wave naturalistic experimental design: pre-, during, and post-summer 2018. During and to some extent post-summer 2018, we find significant increases in participants failing response validity indicators, decreases in reliability and validity of a widely used personality measure, and failures to replicate well-established findings. However, these detrimental effects can be mitigated by using response validity indicators and screening the data. We discuss implications and offer suggestions to ensure data quality.

Subfields: Applied Psychology, Computer Science Applications, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Computer Science Applications

In Evaluation & Benchmarks

Citations in the corpus

ChatGPT outperforms crowd workers for text-annotation tasks (2023, background): “Moreover, there have been concerns that MTurk data quality has decreased (Chmielewski and Kucker, 2020), while alternative platforms such as CrowdFlower and FigureEight are no longer practicable options for academic research since they were acquired by Appen, a company that is focused on a business…”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “[25] Michael Chmielewski and Sarah C Kucker.”
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA (2023, background): “, Cronbach’s alphas and internal validity between scales) [2].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Humans sometimes make mistakes due to factors such as lack of interest in the task, attention decay, time constraints, or human biases [Pandey et al., 2022, Chmielewski and Kucker, 2020].”
Towards Understanding Sycophancy in Language Models (2023, neutral): “Human evaluators are imperfect (Saunders et al., 2022; Gudibande et al., 2023), make mistakes e.g., due to limited time (Chmielewski & Kucker, 2020) or cognitive biases (Pandey et al., 2022), and sometimes have diverse, contradictory preferences (Bakker et al., 2022).”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, background): “Artifacts can also occur from poor-quality annotations and less attentive annotators, both of which are on the rise on crowdsourcing platforms (Chmielewski and Kucker, 2020; Arechar and Rand, 2020).”