Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk

By Tanushree Mitra, Clayton J. Hutto, Eric Gilbert. 2015

110 citations recorded by the source corpus.

Subfields: Computer Science Applications, Computer Vision and Pattern Recognition, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Computer Vision and Pattern Recognition

In Safety Research

Citations in the corpus

Evaluation of Text Generation: A Survey (2020, background): “While online crowd-sourcing platforms such as Amazon Mechanical Turk have enabled researchers to run experiments on a larger scale at a lower cost, they come with their own problems, such as maintaining quality control (Ipeirotis et al., 2010; Mitra et al., 2015).”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, methodology): “Such a pre-task filtering approach, sometimes referred to as a person-oriented strategy, is known to outperform process-oriented strategies that perform post-task data filtering or processing [40].”
Empath: Understanding Topic Signals in Large-Scale Text (2016, background): “egory of 200 words, we would have 200=20 = 10 tasks, which must be completed by three workers, at a total cost of 1030:14 = $4:2 for this category. We limit tasks to Masters workers to ensure quality [26] and we aggregate crowdworker feedback by majority vote. If two of three workers believe a word ”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Accuracy (2016, background): “While we explored processcentric measures like varying the acceptance or transparency threshold, previous work has experimented with varying financial incentives [41].”
A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, background): “…all require time and manpower, especially for the evaluation tasks that can only be handled by domain experts; (2) Maintaining quality control [52, 90] — although the emergence of online crowd-sourcing platforms has eased the cost problem to a certain extent, the quality of personnel conducting…”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality (2016, background): “While we explored processcentric measures like varying the acceptance or transparency threshold, previous work has experimented with varying financial incentives [42].”
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets (2017, background): “Prevention techniques include screening for crowdworkers capable of different tasks [18, 27], pre-training crowdworkers [19], maintaining quality while controlling for cost via dynamic task allocation [52, 8], or designing interfaces or payment structures to motivate good work [43, 47, 22].”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, methodology): “When ﬁltering crowd workers, the ﬁrst approach focuses on the pre-qualiﬁcation tasks designed based on the task characteristics (Mitra et al., 2015).”
Rehumanized Crowdsourcing: A Labeling Framework Addressing Bias and Ethics in Machine Learning (2019, background): “…related to dehumanization effects in crowdsourcing have been observed and addressed in prior works, including underpaid contributors (e.g., [13, 14, 20, 30, 40]), incompatible tasks (e.g., [4, 7, 15–17, 20, 21, 36]), tedious work (e.g., [12, 15, 29, 35]), and power imbalance (e.g., [32, 41]).”