Quality management on Amazon Mechanical Turk

By Panagiotis G. Ipeirotis, Foster J. Provost, Jing Wang. AAAI Conference on Human Computation & Crowdsourcing, 2010

1,107 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications

View Paper: https://www.semanticscholar.org/paper/8d4594b4d4827f44b57863376d54536112b7aaca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Quantification of YouTube QoE via Crowdsourcing (2011, background): “Finally, Section VI concludes this work, discussing the potential of the crowdsourcing method.”
Human computation: a survey and taxonomy of a growing field (2011, methodology): “Ipeirotis used expectation maximization to estimate the quality, and also infer some characteristics of the types of errors [29].”
Crowdsourced Data Management: A Survey (2016, background): “Worker elimination, which eliminates low-quality workers or spammers based on worker models, is a very common strategy to improve quality [58], [78], [99].”
HumorTools : A Microtask Workflow for Writing News Satire (2015, methodology): “Some approaches use AI techniques such expectation maximization [28, 14] and decision theoretic control [8].”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “, the EM-like [66, 100] algorithms that these papers use lead to locally optimal solutions.”
Shepherding the crowd yields better work (2012, background): “Requesters can also increase quality by aggregating answers from multiple workers [2,16,23], filtering spammers by inserting test problems with known solutions [19,37], or using intelligent redundancy to simultaneously filter and aggregate [18].”
Scalable multi-label annotation (2014, background): “While the problem of acquiring one label has been well studied [20, 9, 22, 19, 16, 5], to our knowledge the challenge of largescale multi-label annotation has not been addressed before.”
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets (2017, background): “Not only can this lead to poor quality labels and machine learning models with low accuracy, efforts to detect poor quality work (e.g., [25, 10, 22]) in these cases can actually be harmful due to rejection of honest work.”
Learning from the Wisdom of Crowds by Minimax Entropy (2012, methodology): “This assumption underlies the work of Dawid and Skene [5, 18, 19, 11, 17], where each worker is associated with a probabilistic confusion matrix that generates her labels.”
Crowdsourced enumeration queries (2013, background): “…spelling mistakes must be fixed, duplicates must be removed, etc. Similar issues arise in data ingest for traditional database systems through ETL (Extract, Transform and Load) and data integration, but techniques have also been developed specifically for crowdsourced input [5], [6], [7], [8].”
Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk (2015, background): “Ipeirotis, Provost, & Wang [15] use more sophisticated algorithms, which account for both per-item classification error and per-worker biases, to help manage data quality subsequent to data annotation.”
Deep learning from crowds (2017, methodology): “Similarly, Ipeirotis et al. [8] propose using Dawid and Skene’s approach to extract a single quality score for each worker that allows to prune low-quality workers.”
Amazon Mechanical Turk: A Research Tool for Organizations and Information Systems Scholars (2012, positive): “Ipeirotis, Provost, & Wang [20] developed a more sophisticated model for detecting quality of Turkers on a classification task that requires 5 labels per object and 20-30 objects per Turker.”
CDAS: A Crowdsourcing Data Analytics System (2012, background): “Ipeirotis et al. [9] presented a scheme to rank the qualities of workers while Ghosh et al. [7] tried to accurately identify abusive content.”
Counting with the Crowd (2012, methodology): “The approach to getting an item’s label (e.g., identifying a picture of a ﬂower as being blue ) through multiple crowd workers is well-studied [14, 15].”
Machine learning for targeted display advertising: transfer learning in action (2013, background): “For webpage classiﬁcation for safe advertising, data on the very rare classes of interest are drawn from all manner of sources different from the actual target task [3,2,11].”
Social Cues, Social Biases: Stereotypes in Annotations on People Images (2018, background): “While there is signiﬁcant noise in the answers provided by individual workers (Ipeirotis, Provost, and Wang 2010), assigning a HIT to a large number of workers – or exploiting “crowd wisdom” – is often used as a means to mitigate this noise (Kamar, Hacker, and Horvitz 2012).”