The shape of and solutions to the MTurk quality crisis

By Ryan Kennedy, Scott Clifford, Tyler J. Burleigh, Philip D. Waggoner, Ryan Jewell, Nicholas J. G. Winter. Political Science Research and Methods, 2020

541 citations recorded by the source corpus.

Abstract Amazon's Mechanical Turk is widely used for data collection; however, data quality may be declining due to the use of virtual private servers to fraudulently gain access to studies. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. We first analyze 38 studies and show that this fraud is not new, but has increased recently. We then show that these fraudulent respondents provide particularly low-quality data and can weaken treatment effects. Finally, we provide two solutions: an easy-to-use application for identifying fraud in the existing datasets and a method for blocking fraudulent respondents in Qualtrics surveys.

Subfields: Artificial Intelligence, Computer Science Applications, Safety Research

View Paper: https://www.semanticscholar.org/paper/2945a19cc8913bb6acd572838ead115ae93d4b72

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Safety Research

Citations in the corpus

An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results (2019, background): “Even if the current crisis passes, similar issues may arise again (Kennedy et al., 2018).”
Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents (2019, background): “While tools to circumvent the problem were very quickly produced (Kennedy et al., 2018; Ahler et al., 2018), the episode underlined the dangers inherent in overreliance on any one source of subjects.”
The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers (2020, background): “(2019). The Turker blues: Hidden factors behind increased depression rates among Amazon’s Mechanical Turkers.”
Why are conservatives less concerned about the coronavirus (COVID-19) than liberals? Comparing political, experiential, and partisan messaging explanations (2020, methodology): “samples for research related to political ideology (see, e.g., Clifford et al., 2015; Conway et al., 2018; Kennedy et al., 2020).”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “Thus, researchers rating English text should restrict their tasks to English-speaking countries, although Kennedy et al. (2020) find that many workers use Virtual Private Networks (VPNs) to take part in tasks restricted to those in the US.”