Get another label? improving data quality and data mining using multiple, noisy labelers

By Victor S. Sheng, Foster J. Provost, Panagiotis G. Ipeirotis. 2008

1,321 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

ImageNet Large Scale Visual Recognition Challenge (2014, background): “Works such as (Welinder et al., 2010; Sheng et al., 2008; Vittayakorn and Hays, 2011) describe quality control mechanisms for this marketplace.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “Similar claims were made in other communities, stating that, while not as high-quality, overall data quality can actually be improved by having more redundant annotations (Sheng et al., 2008).”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “However, it is tricky to determine the number of ratings to collect for a synset [68]; more ratings lead to a more stable estimate but cost more.”
Why Amazon's Ratings Might Mislead You: The Story of Herding Effects (2014, neutral): “Indeed, models on collective intelligence, from majority voting to collaborating ﬁltering 9–12 and crowdsourcing, 13,14 all assume independent crowds, representing a critical gap between modeling frameworks and empirical insights.”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “[119] shows that, if the labels are noisy,”
Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow (2018, background): “However, the statistical models that power many such applications are only as useful as the data they are trained on, i.e. , garbage in, garbage out [35].”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Crowdsourced datasets are often subject to noise attributed to incorrect labeled annotations (Sheng et al., 2008; Krishna et al., 2016; Ekambaram et al., 2017), which may lead to training models that are not representative of the task at hand.”
YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video (2017, methodology): “This can be partly curbed through replication [37, 44], as we did in stage 2.”
Best of both worlds: Human-machine collaboration for object annotation (2015, methodology): “Approaches such as [10, 46, 28, 62] use iterative improvement to perform a task with accuracy per unit of human cost.”
Active Learning from Crowds (2011, background): “The closest approach to active learning in this area has been the use of repeated labeling (Smyth et al., 1995; Donmez & Carbonell, 2008; Sheng et al., 2008).”
Collaboratively crowdsourcing workflows with turkomatic (2012, background): “Artificial intelligence researchers emphasized the utility of crowds for supporting active learning problems [17,19].”
Comparing Person- and Process-centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk (2015, background): “Here again, the emergence of micro-labor markets has provided a feasible alternative for acquiring large quantities of manual annotations at relatively low cost and within a short period of time—along with several researchers investigating ways to improve the quality of the annotations from inexpert”
crowdsourcing : rating annotators and obtaining cost-effective labels (2010, neutral): “[5] also modeled annotator quality, and showed how repeated and selective labeling increased the overall labeling quality on synthetic data.”