Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment

By Christoph Bartneck, Andreas Duenser, Elena Moltchanova, Karolina Zawieska. PLoS ONE, 2015

296 citations recorded by the source corpus.

Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.

Subfields: Applied Psychology, Computer Science Applications

View Paper: https://www.semanticscholar.org/paper/3ccb172f896f1b523ecf25dc59880d704af90d3e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Computer Science Applications

Citations in the corpus

Beyond confidence: Development of a measure assessing the 5C psychological antecedents of vaccination (2018, background): “While in developed countries, online studies may not be a problem and include people from various educational backgrounds [91], replicating such studies in developing countries may lead to an oversampling of educated participants and therefore limits the generalizability.”
An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk (2017, background): “…the number of studies comparing MTurk samples to student and professional panel samples in domains outside of advertising is increasing (e.g., Bartneck et al. 2015; Berinsky, Huber, and Lenz 2012; Casler, Bickel, and Hackett 2013; Clifford, Jewell, and Waggoner 2015; Goodman, Cryder, and…”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “…2015), interrater reliability within the sample (Conley and Tosti-Kharas, 2014; Costa-jussà et al., 2014), interrater agreement within the sample (Bartneck et al., 2015), and interrater reliability with experts (Alonso and Mizzaro, 2012; Conley and Tosti-Kharas, 2014; Costa-jussà et al., 2014).”
Can we control it? Autonomous robots threaten human identity, uniqueness, safety, and resources (2017, result): “…platform to recruit participants as MTurk workers were found in previous studies to be more representative for the USA population than either student based studies or Internet recruited samples commonly used in other experiments (Bartneck et al., 2015; Buhrmester et al., 2011; Ramsey et al., 2016).”