Vox Populi: Collecting High-Quality Labels from a Crowd

By Ofer Dekel, Ohad Shamir. 2009

166 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Decision Tree Design for Classification in Crowdsourcing Systems (2018, background): “To improve classification performance in crowdsourcing systems, most of the works in the literature focus on enhancing the quality of individual tests, by designing fusion rules to combine decisions from heterogeneous workers [1]–[4], [6], [7], and by investigating the assignment of different tests ”
Artificial Intelligence and Collective Intelligence (2014, methodology): “Instead of bootstrapping the learning process by using multiple workers to redundantly answer each question, Dekel and Shamir (Dekel and Shamir 2009b; 2009a) devise algorithms to limit the influence of bad workers.”
On truth discovery in social sensing: A maximum likelihood estimation approach (2012, methodology): “applied a classification technique to simulate aggregate labels and prune low-quality labelers in a crowd to improve the label quality of the training dataset [9].”
Data-centric Artificial Intelligence: A Survey (2023, unknown): “Other studies focus on algorithmically improving label quality, e.g., pruning low-quality teachers [53].”
Aggregating crowdsourced binary ratings (2013, background): “[4] show that such aggregation can be improved if the bad raters are pruned.”
Learning from the Wisdom of Crowds by Minimax Entropy (2012, background): “1 Introduction There is an increasing interest in using crowdsourcing to collect labels for machine learning [19, 6, 21, 17, 20, 10, 13, 12].”
Asking the Right Questions in Crowd Data Sourcing (2012, methodology): “Machine Learning techniques were also proposed as methods for identifying and pruning out low-quality teachers that are available for an online learning system that is trained on a data set and thus improve its quality [7].”