Demographics and Dynamics of Mechanical Turk Workers

By Djellel Eddine Difallah, Elena Filatova, Panagiotis G. Ipeirotis. 2018

519 citations recorded by the source corpus.

Subfields: Computer Science Applications, Epidemiology, Transportation, Human-Computer Interaction, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Epidemiology

In Transportation

In Human-Computer Interaction

In Social Sciences & Humanities

Citations in the corpus

StereoSet: Measuring stereotypical bias in pretrained language models (2020, background): “is predominantly (80%) below the age of 50 (Difallah et al., 2018), StereoSet may not reflect the stereotypes of the wider US population.”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “This allows us to scale the annotation and obtain ratings from a diverse pool of workers [21, 63], but also poses challenges in simplifying the instructions and in implementing robust quality control.”
Evaluation of Text Generation: A Survey (2020, background): “, 2009), and the demographics of the evaluators may be heavily skewed depending on user base of the platform (Difallah et al., 2018; Reinecke & Gajos, 2015).”
On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current Practices (2021, background): “6% did not report over what timeframe the experiment run, which is known to determine the characteristics of the active pool of workers [19].”
The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems (2022, background): “workers were most likely male, in their early 20s or 30s, and married, with at least one child (Difallah et al., 2018).”
The Expertise Involved in Deciding which HITs are Worth Doing on Amazon Mechanical Turk (2021, background): “Due in large part to this low pay and unfair market, there is a lot of turnover and there are many new Turkers onboarding everyday, by some reports, as many as tens of thousands each day [9], meaning that there is a constant stream of novice Turkers learning the difficult task of finding HITs that a”
Demographic Stability on Mechanical Turk Despite COVID-19 (2020, unknown): “Demographic data showing who used MTurk before COVID-19 [3,4] compares favorably with data from today.”
Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics (2018, background): “To gain insights into the global crowd workforce, it is important to analyze microtask platforms other than MTurk, as the vast majority of MTurk’s workforce consists of workers located in the United States and India (see, e.g., Difallah et al., 2018).”