The online laboratory: conducting experiments in a real labor market

By John Joseph Horton, David G. Rand, Richard J. Zeckhauser. arXiv.org, 2010

1,633 citations recorded by the source corpus.

Online labor markets have great potential as platforms for conducting experiments. They provide immediate access to a large and diverse subject pool, and allow researchers to control the experimental context. Online experiments, we show, can be just as valid—both internally and externally—as laboratory and field experiments, while often requiring far less money and time to design and conduct. To demonstrate their value, we use an online labor market to replicate three classic experiments. The first finds quantitative agreement between levels of cooperation in a prisoner’s dilemma played online and in the physical laboratory. The second shows—consistent with behavior in the traditional laboratory—that online subjects respond to priming by altering their choices. The third demonstrates that when an identical decision is framed differently, individuals reverse their choice, thus replicating a famed Tversky-Kahneman result. Then we conduct a field experiment showing that workers have upward-sloping labor supply curves. Finally, we analyze the challenges to online experiments, proposing methods to cope with the unique threats to validity in an online setting, and examining the conceptual issues surrounding the external validity of online results. We conclude by presenting our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

Subfields: Computer Science Applications, Safety Research

View Paper: https://www.semanticscholar.org/paper/2b6fe8e20f085270cab91760c610c70f40923ab9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Safety Research

Citations in the corpus

Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, methodology): “The opportunity to learn new skills is a tertiary but still important motivating factor (Behrend et al. 2011, Horton et al. 2011, Paolacci et al. 2010) that is probably of particular relevance to tasks related to translation and transcription rather than participation in research studies.”
Monopsony in Online Labor Markets (2020, methodology): “Horton et al. (2011) probability of accepting offer Reward 0.”
Public Attitudes Toward Cognitive Enhancement (2014, neutral): “[51–57].”
What's the right thing to do? Increasing pro-sociality with simple moral nudges (2017, neutral): “We recruit N = 300 subjects living in the US using the online labour market Amazon Mechanical Turk (AMT).”
Statistical physics of human cooperation (2017, background): “Moreover, evidence demonstrates that, across a wide range of economic games, game play is very similar on Amazon Mechanical Turk and in the physical laboratory — even though stakes are generally an order of magnitude higher in the lab [136,137].”
Conducting interactive experiments online (2017, positive): “one-shot Prisoner’s dilemma) conducted with MTurk participants have produced results comparable to those obtained from laboratory samples (e.g. Paolacci et al. 2010; Horton et al. 2011; Goodman et al. 2013).”
Feelings that Make a Difference: How Guilt and Pride Convince Consumers of the Effectiveness of Sustainable Consumption Choices (2014, background): “Several studies demonstrate the suitability of this website for experimental and survey research (Buhrmester et al. 2011; Horton et al. 2011).”