Conducting behavioral research on Amazon’s Mechanical Turk

By Winter A. Mason, Siddharth Suri. 2011

3,324 citations recorded by the source corpus.

Subfields: Computer Science Applications, Information Systems, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Sociology and Political Science

Citations in the corpus

Manipulating and Measuring Model Interpretability (2018, background): “Multiple studies have shown that data from high-reputation Turkers is comparable to data from commercial panels and university pools when assessing outcomes such as attentiveness, honesty, and effort [14, 16, 21, 75, 84].”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Early approaches, such as McKeown (1985), often relied on hand-crafted, domain-dependent structuring rules (which McKeown called schemata).”
Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation (2015, background): “When dealing with crowdsourcing, there is a growing literature on observing and analyzing workers’ behaviour (Mason and Suri 2012) for ultimately being able to detect and eliminate spam (Bozzon et al.”
Swiping me off my feet (2018, methodology): “Mturk provides similar samples to other recruitment methods offering affordability with access to a large, diverse pool (Mason & Suri, 2012).”
An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use (2018, background): “Several authors suggest creating a priori rules about data inclusion/exclusion, carefully tracking attrition rates, and understanding how attrition can influence study results as well as the types of analyses that should be conducted (Mason & Suri, 2012; Zhou et al., 2016).”
Who is Mturk? Personal characteristics and sample consistency of these online workers (2018, background): “…noted demographic differences between MTurkers and the US population on age (Bohannon, 2016; Horton & Chilton, 2010; Mason & Suri, 2011) and gender (Mason & Suri, 2011); between MTurkers and internet samples on age (Buhrmester et al., 2011); and between MTurkers and a US college sample on age and…”
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies (2017, neutral): “A number of studies use “catch trials” and/or longitudinal samples to verify respondent demographics and attention to the tasks at hand (Mason and Suri 2012; Rand 2012).”
psiTurk: An open-source framework for conducting replicable behavioral experiments online (2015, background): “Mason and Suri (2012) provide a general overview of AMT features of interest to academic researchers, one of which is that it provides a way for researchers to easily recruit and compensate individuals for online experiments.”
Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis (2016, methodology): “With these concerns in mind, we took several steps (described in more detail in the supplementary materials online), consistent with established best practices (Mason and Suri 2012), to ensure high quality ratings.”
Using Mechanical Turk to Study Clinical Populations (2013, neutral): “They are paid by requesters upon successful completion of tasks (for an introduction to using MTurk, see Mason & Suri, 2012).”
Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “…attractive to researchers: A shared data security and payment infrastructure is provided by Amazon, which lowers fixed overhead costs, reduces the need for technical expertise or administrative support, and avoids payment hassles that may disincentivize respondent participation (Mason & Suri 2012).”
Computational Analysis of Persuasiveness in Social Multimedia: A Novel Dataset and Multimodal Prediction Approach (2014, methodology): “1 Subjective Annotations Amazon Mechanical Turk (AMT) [21], which is a popular online crowdsourcing platform, was used to obtain subjective evaluations of the speaker in each video.”