A reliability analysis of Mechanical Turk data

By Steven V. Rouse. Computers in Human Behavior, 2015

409 citations recorded by the source corpus.

Subfields: Computer Science Applications, Management Information Systems, Safety Research

View Paper: https://www.semanticscholar.org/paper/d3b1baff95315ecb80fafd81f7b8e3efee65bb67

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Management Information Systems

In Safety Research

Citations in the corpus

Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “As many commentators point out, careless or inattentive responding results in error variance (or noise; Goodman et al., 2012; Fleischer et al., 2015; Huang et al., 2015b; McGonagle, 2015; Ran et al., 2015; Rouse, 2015).”
Why Do Adults Engage in Cyberbullying on Social Media? An Integration of Online Disinhibition and Deindividuation Effects with the Social Structure and Social Learning Model (2016, background): “…following procedural remedies taken from the literature (e.g., Goodman et al. 2013; Landers & Behrend 2015; Lowry et al. 2016; Lowry et al. 2013; Rouse 2015; Steelman et al. 2014) that have been shown to address these issues: (1) We randomized the order of the survey questions; (2) we reversed…”
Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments (2017, background): “…science (Berinsky et al., 2012, 2013; Clifford, Jewell, & Waggoner, 2015; Mullinix, Leeper, Druckman, & Freese, 2015).1 The psychometric properties of multiple well-established personality scales have even been quantitatively replicated (Buhrmester et al., 2011; Peer et al., 2013; Rouse, 2015).”
Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA (2023, methodology): “For example, in testing data quality on MTurk, Rouse [35] used the openness subscale of a Big-5 Personality Inventory measure and compared the alpha value for the MTurk sample with the alpha value found by Goldberg [36] who used a large-scale national sample.”
"Cargo Cult" science in traditional organization and information systems survey research: A case for using nontraditional methods of data collection, including Mechanical Turk and online panels (2016, background): “It is dramatically less expensive and time consuming, and it enables a wider range of scientists and students to access research participants (Rouse, 2015) There could be data quality issues.”
MTurk Research: Review and Recommendations (2020, background): “Accordingly, to address concerns about how different challenges may threaten validity of results obtained and conclusions reached when using MTurk (Hydock, 2018; Rouse, 2015), there is a need to clearly describe all steps (Thomas & Clifford, 2017; Zhou & Fishbach, 2016).”
The Future of Work in the ‘Sharing Economy’. Market Efficiency and Equitable Opportunities or Unfair Precarisation? (2016, neutral): “…experiments, and user studies; as a result there is a growing body of research discussing the reliability and validity of this platform for research purposes (Grysman, 2015; Kittur et al., 2008; Layman & Sigurdsson, 2013; Paolacci & Chandler, 2014; Rand, 2012; Rouse, 2015; Schmidt, 2015) xl .”
How serious is the ‘carelessness’ problem on Mechanical Turk? (2019, background): “We join with others (Bates & Lanza, 2013; Lian et al., in press; Oppenheimer et al., 2009; Rouse, 2015) in recommending the use of validity checks in survey research, especially for data collected through MTurk.”
How psychological and contextual factors contribute to travelers’ propensity to choose green hotels? (2019, methodology): “We carefully followed the measures/ steps recommended to ensure the quality and reliability of the data collection procedure using MTurk (Hauser and Schwarz, 2016; Rouse, 2015).”