Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk

By Adam J. Berinsky, G. Huber, Gabriel S. Lenz. 2012

4,015 citations recorded by the source corpus.

Subfields: Computer Science Applications, Economics and Econometrics, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Economics and Econometrics

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021, methodology): “The AMT evaluators mentioned grammar and spelling far more frequently than the expert evaluators, who were more likely to mention the repetition, factuality, and commonsense of the passage.”
Worth Weighting? How to Think About and Use Weights in Survey Experiments (2017, background): “Other firms and platforms, such as Survey Sampling International, Google Consumer Surveys (Santoso, Stein, and Stevenson 2016), and Amazon’s Mechanical Turk (Berinsky, Huber, and Lenz 2012), offer even less costly access to large anddiverse convenience samples onwhich researchers can also conduct su”
Coronavirus Fears and Macroeconomic Expectations (2020, positive): “1Mechanical Turk allows for recruitment of subject pools that are more nationally representative than typical convenience samples, making it a popular choice for social science experiments (Berinsky et al., 2012; Casler et al., 2013; Levay et al., 2016).”
An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk (2017, background): “…comparing MTurk samples to student and professional panel samples in domains outside of advertising is increasing (e.g., Bartneck et al. 2015; Berinsky, Huber, and Lenz 2012; Casler, Bickel, and Hackett 2013; Clifford, Jewell, and Waggoner 2015; Goodman, Cryder, and Cheema 2013; Heen,…”
Good Housekeeping, Great Expectations: Gender and Housework Norms (2019, background): “Nevertheless, it should be noted that the participants on the platform are slightly more educated and more technologically experienced than the average American (Berinsky, Huber, and Lenz 2012; Buhrmester, Kwang, and Gosling 2011).”
When Terror Hits Home: Identity Fused Americans Who Saw Boston Bombing Victims as “Family” Provided Aid (2015, methodology): “Such perceptions of kinship are potentially important because they are associated with strong feelings of inherent value, protection, obligation, and duty (Bailey, 1988).”
Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants (2015, background): “Furthermore, highly influential MTurk-related methods articles have set a precedent for such restrictions as a way to ease concern over data quality (Berinsky et al., 2012; Goodman et al., 2013; Paolacci et al., 2010).”
Income Inequality Influences Perceptions of Legitimate Income Differences (2017, neutral): “35#TAB##R# Osberg and Smeeding 2006, see also Norton and Ariely 2010 for a similar observation with respect to wealth inequality 36 For a discussion of Mechanical Turk as a subject pool, see Berinsky et al. (2012), Buhrmeister et al.”
Does Counter-Attitudinal Information Cause Backlash? Results from Three Large Survey Experiments (2018, background): “While opt-in samples collected via MTurk should not be considered representative of the US population, they have been shown to outperform laboratory-based convenience samples in ensuring adequate variation across sociodemographic and political characteristics of interest (Berinsky et al. 2012).”