Running experiments on Amazon Mechanical Turk
By Gabriele Paolacci, Jesse J. Chandler, Panagiotis G. Ipeirotis. 2010
4,700 citations recorded by the source corpus.
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.
Subfields: Computer Science Applications, Sociology and Political Science, Transportation, Human-Computer Interaction, Cognitive Science & Psychology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Science Applications
- Jesse J. Chandler
- Gabriele Paolacci
- D. Shapiro
- David G. Rand
- J. Goodman
- Scott Clifford
- Kim Bartel Sheehan
- P. D. Harms
In Sociology and Political Science
- Scott Clifford
- Jesse J. Chandler
- Kyle A. Thomas
- David G. Rand
- Gabriele Paolacci
- D. Shapiro
- Sharad Goel
- John B. Ford
In Transportation
- David G. Rand
- Kimmo Eriksson
- Kristina Rennekamp
- S. Piantadosi
- J. Coultas
- Katherine Burson
- D. Shapiro
- Kyle A. Thomas
In Human-Computer Interaction
- David G. Rand
- Kimmo Eriksson
- Kristina Rennekamp
- S. Piantadosi
- J. Coultas
- Katherine Burson
- Scott Clifford
- D. Shapiro
In Cognitive Science & Psychology
- David G. Rand
- Kimmo Eriksson
- Kristina Rennekamp
- S. Piantadosi
- J. Coultas
- Norbert Schwarz
- Katherine Burson
- D. Shapiro
Citations in the corpus
- MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text (2013, background): “The average worker is 36 years old, more educated than the United States population in general (Paolacci et al., 2010), and the majority of workers are female.”
- Beyond Goldfish Memory: Long-Term Open-Domain Conversation (2021, background): “Crowdsourcing long conversations is difficult due to both the expense and the difficulty of employing crowdworkers for long lengths of time due to so called Human Intelligence Tasks (HITs) being typically of a short duration – only “a few minutes” (Paolacci et al., 2010).”
- Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis (2016, background): “…that crowd workers reliably replicate the behavior of undergraduate students across a wide variety behavioral experiments (Buhrmester et al., 2011; Paolacci et al., 2010; Berinsky et al., 2012; Mason and Suri, 2012; Goodman et al., 2013), and produce verifiably high-quality work in labeling tasks…”
- A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles (2021, background): “MTurk is an online crowd-sourcing platform for online data collection with a diverse population of workers [16, 75].”
- Introduction to the Special Issue - Social Media and Business Transformation: A Framework for Research (2013, background): “Increasingly, social media are also transforming the way businesses relate to workers, allowing them to build flexi-ble relationships with remote talent (Archak 2010), to crowd source new ideas (Di Gangi and Wasko 2009), or to engage in micro outsourcing (Goldman et al. 2011, Paolacci et al. 2010).”
- Cultural antecedents of green behavioral intent: An environmental theory of planned behavior (2015, result): “Paolacci, Chandler, and Ipeirotis (2010) demonstrated that data collected via Mechanical Turk was comparable to laboratory-based data collection.”
- A Tale of Two Sample Sources: Do Results from Online Panel Data and Conventional Data Converge? (2018, background): “Indeed, evidence suggests that OPD samples are more diverse, younger, more educated, but more poorly paid than the general US population (Paolacci et al., 2010; Gosling et al., 2004; Sprouse, 2011) and, at the same time, more diverse, older, and more work experienced than a typical undergraduate…”
- Human Rights Organizations as Agents of Change: An Experimental Examination of Framing and Micromobilization (2015, background): “In addition, participants’ profiles appear to be consistent with those of other studies using MTurk (Berinsky et al. 2012; Buhrmester et al. 2011; Paolacci et al. 2010), many of which find high levels of internal consistency of self-reported demographics”