How serious is the ‘carelessness’ problem on Mechanical Turk?
By Mara S. Aruguete, H. Huynh, Blaine L. Browne, Bethany S. Jurs, Emilia S. Flint, L. McCutcheon. International Journal of Social Research Methodology, 2019
88 citations recorded by the source corpus.
ABSTRACT This study compared the quality of survey data collected from Mechanical Turk (MTurk) workers and college students. Three groups of participants completed the same survey. MTurk respondents completed the survey as paid workers using the Mechanical Turk crowdsourcing platform. Student Online respondents also completed the survey online after having been recruited in class. Finally, Student Paper-and-Pencil respondents completed the survey on paper in a classroom setting. Validity checks embedded in the survey were designed to gauge participants’ haste and carelessness in survey completion. MTurk respondents were significantly more likely to fail validity checks by contradicting their own answers or simply completing the survey too quickly. Student groups showed fewer careless mistakes and longer completion times. The MTurk sample tended to be older, more educated, and more ethnically diverse than student samples. Results suggest that researchers should pay special attention to the use of validity checks when recruiting MTurk samples.
Subfields: Computer Science Applications, Information Systems
View Paper: https://www.semanticscholar.org/paper/ed68dfd2236ccb59470fd0e924ac9ed55a0a1d24
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Science Applications
In Information Systems
Citations in the corpus
- An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results (2019, result): “However, the percentage of participants failing validity indicators in Waves 3 and 4 is similar to informal reports and recent publications (Aruguete et al., 2019; Courrégé, Skeel, Feder, & Boress, 2019; Dreyfuss, 2018).”