Data quality of platforms and panels for online behavioral research

By Eyal Péer, David M. Rothschild, Andrew Gordon, Zak Evernden, Ekaterina Damer. Behavior Research Methods, 2021

1,414 citations recorded by the source corpus.

We examine key aspects of data quality for online behavioral research between selected platforms (Amazon Mechanical Turk, CloudResearch, and Prolific) and panels (Qualtrics and Dynata). To identify the key aspects of data quality, we first engaged with the behavioral research community to discover which aspects are most critical to researchers and found that these include attention, comprehension, honesty, and reliability. We then explored differences in these data quality aspects in two studies (N ~ 4000), with or without data quality filters (approval ratings). We found considerable differences between the sites, especially in comprehension, attention, and dishonesty. In Study 1 (without filters), we found that only Prolific provided high data quality on all measures. In Study 2 (with filters), we found high data quality among CloudResearch and Prolific. MTurk showed alarmingly low data quality even with data quality filters. We also found that while reputation (approval rating) did not predict data quality, frequency and purpose of usage did, especially on MTurk: the lowest data quality came from MTurk participants who report using the site as their main source of income but spend few hours on it per week. We provide a framework for future investigation into the ever-changing nature of data quality in online research, and how the evolving set of platforms and panels performs on these key aspects.

Subfields: Applied Psychology, Experimental and Cognitive Psychology

View Paper: https://www.semanticscholar.org/paper/9bd4a36cc6a53c08976978381d1d7eca3c8aab11

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Experimental evidence on the productivity effects of generative artificial intelligence (2023, background): “In a preregistered online experiment, we recruited 453 experienced, college-educated professionals on the survey platform Prolific and assigned each to complete two occupation-specific, incentivized writing tasks (21).”
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA (2023, background): “A recent controversy arose between researchers affiliated with Prolific and CloudResearch when individuals affiliated with Prolific (including the company’s CEO) published findings stating that Prolific offered the highest quality data among a series of data collection platforms including CloudResea”