MTurk Character Misrepresentation: Assessment and Solutions

321 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Manipulating and Measuring Model Interpretability (2018, background): “3Although recent research has shown that Turkers may manipulate their demographic information so as to be included in studies [96], we only screened Turkers using two criteria, both of which are enforced by the platform and would require some effort to manipulate: country and approval rating.”
MTurk Research: Review and Recommendations (2020, background): “…pay everyone who participates, eliminate those who do not match desired criteria, and invite those who meet the qualifications/pass the screener to participate in the focal study (Chandler, Mueller, & Paolacci, 2014; Hydock, 2018; Siegel, Navarro, & Thomson, 2015; Wessling, Huber, & Netzer, 2017).”
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk (2018, background): “Wessling, Huber, and Netzer (2017) disagreed, identifying an imposter rate of 24 – 83% across a series of MTurk studies.”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “This is most likely due to the fact that most requesters are reluctant to reject HITs regardless of quality, which results in an estimated 95% of workers having an approval rate of 98% or above (Matherly, 2019; Wessling et al., 2017).”