Running experiments on Amazon Mechanical Turk

By Gabriele Paolacci, Jesse J. Chandler, Panagiotis G. Ipeirotis. 2010

4,700 citations recorded by the source corpus.

Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.

Subfields: Computer Science Applications, Sociology and Political Science, Transportation, Human-Computer Interaction, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

  1. Jesse J. Chandler
  2. Gabriele Paolacci
  3. D. Shapiro
  4. David G. Rand
  5. J. Goodman
  6. Scott Clifford
  7. Kim Bartel Sheehan
  8. P. D. Harms

In Sociology and Political Science

  1. Scott Clifford
  2. Jesse J. Chandler
  3. Kyle A. Thomas
  4. David G. Rand
  5. Gabriele Paolacci
  6. D. Shapiro
  7. Sharad Goel
  8. John B. Ford

In Transportation

  1. David G. Rand
  2. Kimmo Eriksson
  3. Kristina Rennekamp
  4. S. Piantadosi
  5. J. Coultas
  6. Katherine Burson
  7. D. Shapiro
  8. Kyle A. Thomas

In Human-Computer Interaction

  1. David G. Rand
  2. Kimmo Eriksson
  3. Kristina Rennekamp
  4. S. Piantadosi
  5. J. Coultas
  6. Katherine Burson
  7. Scott Clifford
  8. D. Shapiro

In Cognitive Science & Psychology

  1. David G. Rand
  2. Kimmo Eriksson
  3. Kristina Rennekamp
  4. S. Piantadosi
  5. J. Coultas
  6. Norbert Schwarz
  7. Katherine Burson
  8. D. Shapiro

Citations in the corpus