Demographic Stability on Mechanical Turk Despite COVID-19

By Aaron J. Moss, Cheskie Rosenzweig, Jonathan Robinson, Leib Litman. 2020

72 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures (2021, background): “While there is research showing that Amazon Mechanical Turk workers show some reasonable amount of diversity (Moss et al., 2020), this is still a narrow, US-centric set.”
Perturbation Augmentation for Fairer NLP (2022, background): “The demographics of our workers roughly matches a recent survey of AMT workers (Moss et al., 2020), which found that workers skew white, and are more likely to be women than men.”
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback (2022, background): “While these workers are reasonably diverse (Moss et al., 2020), this is quite different to a public deployment with organic users, who are using the system not because they are being paid but because they are genuinely engaged.”
Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls (2022, background): “While careful instructions (Huynh et al., 2021) can result in good quality feedback or labels to learn from, collection both involves signiﬁcant monetary costs – where anno-tators should be paid well above minimum wage – and the pool of workers may be limited (Moss et al., 2020).”