Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

By Eyal Péer, Joachim Vosgerau, Alessandro Acquisti. Behavior Research Methods, 2013

1,615 citations recorded by the source corpus.

Subfields: Computer Science Applications, Safety Research, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/9fffaf5e8a1699ed6dd4484bb272013a67caf3bd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Safety Research

In Sociology and Political Science

Citations in the corpus

When Victims Help Their Abusive Supervisors: The Role of LMX, Self-Blame, and Guilt (2020, background): “%) of approvedversus-submitted tasks (Hauser & Schwarz, 2015; Litman, Robinson, & Rosenzweig, 2014; Peer, Vosgerau, & Acquisti, 2014) because they have been found to be more attentive in online tasks (Goodman et al., 2012; Peer et al., 2014).”
Data quality of platforms and panels for online behavioral research (2021, positive): “…reputation (approval ratings — share of participants ’ submissions that were accepted by the researcher) can positively predict higher data quality, as high reputation was found to produce better data quality, to the point that even attention-check questions were unnecessary (Peer et al., 2014).”
MTurk Research: Review and Recommendations (2020, background): “…familiarity with common manipulations, attention check techniques, and experimental tasks and questions; Lovett, Bajaba, Lovett, & Simmering, 2018; Peer, Vosgerau, & Acquisti, 2014) or, alternatively, employ screening questions to gauge MTurker familiarity with research subject, stimuli, and if…”
How Black Are Lakisha and Jamal? Racial Perceptions from Names Used in Correspondence Audit Studies (2017, positive): “Additionally, Peer, Vosgerau, and Acquisti (2014) find that limiting HITs to high-reputation workers (those with ≥95 percent HIT approval ratings) is sufficient to maintain high data quality, and adding catch trials or attention-check questions do not improve data quality further.”
Advantages of Matched Over Unmatched Opt-in Samples for Studying Criminal Justice Attitudes: A Research Note (2020, background): “Respondents were limited to those who had completed over 500 HITs (human intelligence tasks), had a 95% or higher approval rating, and were aged 18 or older in order to capture high-quality respondents (Peer et al., 2014).”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “Past research suggests that MTurk samples may be prone to inattention (Chandler et al., 2014), social desirability bias (Behrend et al., 2011; Antin and Shaw, 2012), and dishonesty (Rand, 2012; Shapiro et al., 2013; Peer et al., 2014; Daly and Nataraajan, 2015).”
Recruiting Participants With Programming Skills: A Comparison of Four Crowdsourcing Platforms and a CS Student Mailing List (2022, background): “Other studies measure the quality of work in crowdsourcing platforms such as MTurk, Prolific, and CrowdFlower [44, 45].”
The scales of general well-being (SGWB) (2017, background): “Once again, only individuals whose previous work had been rated as adequate more than 95% of the time were able to access the survey (Peer et al., 2014).”
Conducting Clinical Research Using Crowdsourced Convenience Samples. (2016, background): “HAR alone does an excellent job discriminating attentive from inattentive workers (Peer et al. 2014).”