Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants

By David J. Hauser, Norbert Schwarz. Behavior Research Methods, 2015

1,698 citations recorded by the source corpus.

Subfields: Computer Science Applications, Sociology and Political Science, Statistical and Nonlinear Physics

View Paper: https://www.semanticscholar.org/paper/17c0197c2002d167524726b88dffc33596f8ccf2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Sociology and Political Science

In Statistical and Nonlinear Physics

Citations in the corpus

Formalizing Neurath’s Ship: Approximate Algorithms for Online Causal Learning (2016, positive): “It offers an well validated (Buhrmester, Kwang, & Gosling, 2011; Crump, McDonnell, & Gureckis, 2013; Gosling, Vazire, Srivastava, & John, 2004; Hauser & Schwarz, 2015; Mason & Suri, 2012) subject pool, diverse in age and background, suitable for high-level cognition tasks.”
How serious is the ‘carelessness’ problem on Mechanical Turk? (2019, background): “On the other hand, some researchers have found no differences between MTurk workers and college student samples on attentiveness (Behrend, Sharek, Meade, & Wiebe, 2011; Casler et al., 2013; Hauser & Schwarz, 2015).”
Are Attention Check Questions a Threat to Scale Validity? (2018, background): “…2005), and this percentage can be as high as 60 per cent when respondents receive little or no incentive to complete a survey (Berry et al., 1992; Hauser & Schwarz, 2016; Meade * Address for correspondence: Franki Kung, Department of Psychology, University of Waterloo, 200 University Ave. W.,…”
An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk (2017, result): “With over 90% of the MTurk workers passing the IMC used in this study, compared to around 50% for the professional samples, our results from this advertising experiment are consistent with the former (Hauser and Schwarz 2016).”
On making experimental design choices: Discussions on the use and challenges of demand effects, incentives, deception, samples, and vignettes (2020, result): “While some studies have reported increased attentiveness of MTurk samples as compared to student samples (Klein et al., 2014; Hauser & Schwarz, 2016), the use of attention checks with any sample is an important experimental protocol (Abbey & Meloy, 2017; Kane & Barabas, 2019).”
Gender-Inclusive Bathrooms Signal Fairness Across Identity Dimensions (2018, background): “Notably, college research pools can be less attentive than Amazon Mechanical Turk (MTurk) samples (Hauser & Schwarz, 2016), and excluding participants who fail attention checks promotes higher quality data (Peer, Vosgerau, & Acquisti, 2014).”
The shape of and solutions to the MTurk quality crisis (2020, positive): “In fact, MTurk samples generally provide higher quality data than student samples, community samples, and even some high-quality national samples (Hauser and Schwarz, 2015; Mullinix et al ., 2015; Thomas and Clifford, 2017; Anson, 2018).”
No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments (2018, background): “In other words, one concern is that an FMC placed after the outcome measure may fail to accurately measure attentiveness during the preceding experimental portion of the survey (e.g., Hauser and Schwarz 2015, 5).”