Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers

By Jesse J. Chandler, Pam Mueller, Gabriele Paolacci. Behavior Research Methods, 2013

967 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications, Safety Research

View Paper: https://www.semanticscholar.org/paper/6314553dd70d7e708e0ed64080709a67b9475e7f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Safety Research

Citations in the corpus

Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants (2015, methodology): “Because of these differences, researchers often worry that MTurk participants are inattentive to instructions and provide poor-quality data (Chandler, Mueller, & Paolacci, 2014).”
psiTurk: An open-source framework for conducting replicable behavioral experiments online (2015, background): “A common worry with AMT is the unrepresen-tativeness of its subject pool (in terms of demographics and geography), as well as the fact that certain experimental protocols are now too well-known to workers to guarantee naive participants (Chandler et al., 2014).”
Conducting interactive experiments online (2017, neutral): “…from revising their decisions during a given interaction. usually moderated and prohibit the dissemination of the content of HITs and the discussion of strategies, typically center upon the attractiveness of a HIT in terms of earnings and length rather than its content (Chandler et al. 2014).”
MTurk Character Misrepresentation: Assessment and Solutions (2017, methodology): “…do not explore the disturbing finding that people who complete many social psychology research studies become non-naı̈ve, and are thus differentially affected by specific manipulations, various forms of memory tasks, and attention checks (Chandler, Mueller, and Paolacci 2014; Chandler et al. 2015).”
How serious is the ‘carelessness’ problem on Mechanical Turk? (2019, background): “Further, he claimed, as did Krantz (2015) and Chandler et al. (2013) that there is no way to guarantee that workers are giving their undivided attention to survey items.”
“Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents (2015, background): “In the last several years Amazon’s Mechanical Turk (MTurk) has surged in popularity in experimental and survey-based social science research (Berinsky et al., 2012; Chandler et al., 2014; Krupnikov and Levine, 2014; Paolacci and Chandler, 2014).”
Dispositional Sources of Managerial Discretion: CEO Ideology, CEO Personality, and Firm Strategies (2018, methodology): “We closely followed the checks (e.g., comprehension and time taken to fill out the survey) recommended to ensure the quality of the responses in MTurk and used only U.S. subjects (Chandler, Mueller, and Paolacci, 2014).”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “With regard to experimental research, MTurk samples are at risk of worker non-naïveté and selective attrition as a function of experimental condition posing a risk for both completeness and the accuracy of inferences drawn (Chandler et al., 2014; Zhou and Fishbach, 2016).”
Headphone screening to facilitate web-based auditory experiments (2017, methodology): “Experimenters in these fields have developed methods to maximize the quality of web-collected data (Meade and Bartholomew, 2012; Chandler et al., 2013).”
Response Speed and Response Consistency as Mutually Validating Indicators of Data Quality in Online Samples (2017, background): “By selecting the repeated items randomly for each participant, this procedure should also reduce the ability of participants to report quality screens to subsequent participants, as often occurs with more direct data quality items (Chandler et al., 2014; Harms & DeSimone, 2015).”
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies (2017, neutral): “MTurk may also slightly underrepresent blacks and Hispanics while overrepresenting Asians (Berinsky et al. 2012; Chandler, Mueller, and Paolacci 2014).”