Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research

By Matthew J. C. Crump, John V. McDonnell, Todd M. Gureckis. 2013

1,587 citations recorded by the source corpus.

Amazon Mechanical Turk (AMT) is an online crowdsourcing service where anonymous online workers complete web-based tasks for small sums of money. The service has attracted attention from experimental psychologists interested in gathering human subject data more efficiently. However, relative to traditional laboratory studies, many aspects of the testing environment are not under the experimenter's control. In this paper, we attempt to empirically evaluate the fidelity of the AMT system for use in cognitive behavioral experiments. These types of experiment differ from simple surveys in that they require multiple trials, sustained attention from participants, comprehension of complex instructions, and millisecond accuracy for response recording and stimulus presentation. We replicate a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT. While most of replications were qualitatively successful and validated the approach of collecting data anonymously online using a web-browser, others revealed disparity between laboratory results and online results. A number of important lessons were encountered in the process of conducting these replications that should be of value to other researchers.

Subfields: Cognitive Neuroscience, Computer Science Applications, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Science Applications

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018, background): “[9] show that AMT can be reliably used to replicate many psychophysics studies, despite the inability to control all environmental factors.”
Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, methodology): “…including the Implicit Association Task (Klein et al. 2014) and Stroop Task (Crump et al. 2013), or that rely on brief presentation of stimuli, such as attentional blink, flanker, and subliminal priming paradigms (Crump et al. 2013), have all been successfully implemented using MTurk samples.”
jsPsych: A JavaScript library for creating behavioral experiments in a Web browser (2014, result): “Although some concerns remain about certain kinds of behavioral experiments, such as cognitive-learning and priming paradigms (Crump et al., 2013), the general consensus seems to be that online experiments produce results similar to those of laboratory experiments for many tasks.”
Disentangling the Effects of Social Signals (2014, methodology): “We studied peer recommendation with randomized experiments on Amazon Mechanical Turk (Mturk), which is a popular platform for experimental behavioral research (Bohannon, 2011; Mason and Suri, 2012; Crump et al., 2013; Kittur et al., 2013).”
How Deep is the Feature Analysis underlying Rapid Visual Categorization? (2016, background): “AMT is a powerful tool that allows the recruitment of massive trials of anonymous workers screened with a variety of criteria [3].”
Inflation Expectations, Learning and Supermarket Prices: Evidence from Survey Experiments (2017, methodology): “We followed several guidelines that describe the best practices for recruiting individuals for online surveys and experiments using AMT in order to ensure high quality responses (see, for instance, Crump et al., 2013).”