Who are the Turkers? Worker Demographics in Amazon Mechanical Turk

By Joel Ross, Andrew Zaldivar, Lilly C. Irani, Bill Tomlinson. 2009

180 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Learning from Dialogue after Deployment: Feed Yourself, Chatbot! (2019, background): “tings with significant differences from the deployment environment (Ross et al., 2009).”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “This allows us to scale the annotation and obtain ratings from a diverse pool of workers [21, 63], but also poses challenges in simplifying the instructions and in implementing robust quality control.”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “…is a significant limitation of this tooling: crowdworker populations may not be representative of the eventual audience of a deployed model (Ross et al., 2009), and in particular, it is important in any human studies to ensure the inclusion of people from underrepresented and marginalized…”
Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics (2018, background): “Early studies on the demographics of workers on MTurk (Ipeirotis, 2010b; Ross et al., 2009, 2010; Paolacci et al., 2010; Kazai et al., 2012) found that the vast majority of workers were located in the USA and India, and that they were young and highly educated.”
Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations (2017, background): “This is consistent with past ﬁndings (e.g., Ross et al., 2010; Behrend et al., 2011).”