Last Words: Amazon Mechanical Turk: Gold Mine or Coal Mine?

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “fair wage? Fort et al. (2011) discuss the low wages MTurk workers receive, along with concerns about data quality issues that the platform incentivizes.”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, background): “We recognize that human annotations collected from crowd workers cannot be considered as an absolute ground truth of social biases as they are influenced by annotator bias such as those arising from the cultural background or demographics of the annotator [12].”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…issue with such platforms concerns their ethical implications (for example, they involve large groups of poorly paid individuals; see Fort et al., 2011) as well as the reliability of the data collected, though measures can be put in place to ensure, for instance, that contributors…”
Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines (2014, background): “The advantages of crowdsourcing over expert-based annotation have already been discussed elsewhere (Fort et al., 2011; Wang et al., 2012), but in a nutshell, crowdsourcing tends to be cheaper and faster.”
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation (2021, background): “Leveraging the wisdom-of-the-crowd (Galton, 1907; Yi et al., 2010) is common in our field of NLP, often through the use of crowdsourcing platforms like Amazon Mechanical Turk that provide access to many raters, although not representative of the broader population (Fort et al., 2011).”
Collection of a Large Database of French-English SMT Output Corrections (2012, background): “Many recent articles have shown the effectiveness of MTurk to Figure 1: Post-edition task interface create annotated data in the ﬁeld of natural language processing but, however, the ethical, social and economic aspects implied by such tools are subject to intense debates (Fort et al., 2011).”
Being a turker (2014, background): “To date much of the research on AMT takes the employers’ perspective, e.g. [14, 15, 17, 18], and this has in turn been highlighted [6, 16].”
Ethical by Design: Ethics Best Practices for Natural Language Processing (2017, background): “Doing the research itself may involve hiring helpers, who should not be kept in unethical work conditions; crowdsourcing has been criticized to be a form of slavery, for instance (Fort et al., 2011).”
Crowdsourcing research: Data collection with Amazon’s Mechanical Turk (2018, background): “A 2011 study estimated that the global active worker pool was between 5059 and 42,912 people (Fort, Adda, & Cohen, 2011).”
Crowdsourcing Consumer Research (2017, background): “…wage, and whereas most evidence suggests that pay rates do not affect data quality in a typical consumer study (Burhmester et al. 2011; Goodman et al. 2013; Mason and Watts 2010; Litman et al. 2015; but see Fort et al. 2011 for an exception), adequate payments are in the researchers’ best interest.”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “While the amount of compensation does not appear to be aﬀected by data quality, we urge researchers to make ethical considerations when determining payment (Bederson and Quinn, 2011; Fort et al., 2011; Bergvall-Kåreborn and Howcroft, 2014).”
Can machine translation systems be evaluated by the crowd alone (2015, background): “Fort, Adda and Cohen (2011) identify as a main cause of low payment to workers the lack of adequate worker reputation systems: without a method of accurately targeting good workers, increased payment simply attracts larger numbers of aggressive optimizers.”
Demographics and Dynamics of Mechanical Turk Workers (2018, background): “Several researchers raise the concern regarding the ethics of using MTurk workers for research experiments [15, 16].”
Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms (2015, background): “It is unlikely that all work can entice volunteers or be gamiﬁed, and unpaid in-game work is not clearly more ethical than paying something (Fort et al., 2011).”