Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise

By J. Whitehill, Paul Ruvolo, Tingfan Wu, J. Bergsma, Javier R. Movellan. Neural Information Processing Systems, 2009

1,336 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/6953420c593842697dd09bc2cf7ffbbaf67a6e8e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Flock: Hybrid Crowd-Machine Learning Classifiers (2015, unknown): “, [8], [21], [47]).”
Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition (2019, methodology): “To get rid of noisy labels, motivated by [73], an Expectation Maximization (EM) framework was used to assess each labeler’s reliability.”
Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection (2015, background): “ers who have adequate expertise for fine-grained classification [36, 38, 28, 22].”
Combining human and machine intelligence in large-scale crowdsourcing (2012, methodology): “apply unsupervised learning to simultaneously predict the correct answer of a task, the difficulty of the task and the accuracy of workers based on some assumptions about the underlying relationships between the answer, the task, and workers [14].”
Error Rate Bounds in Crowdsourcing Models (2013, unknown): “M L ] 1 0 Ju l 2 01 3”
Scalable multi-label annotation (2014, background): “While the problem of acquiring one label has been well studied [20, 9, 22, 19, 16, 5], to our knowledge the challenge of largescale multi-label annotation has not been addressed before.”
Estimating the accuracies of multiple classifiers without labeled data (2014, methodology): “Indeed, both Zhang et al. [2014] and Jain and Oh [2014] derived such methods based on three-dimensional tensors.”
Learning from the Wisdom of Crowds by Minimax Entropy (2012, background): “1 Introduction There is an increasing interest in using crowdsourcing to collect labels for machine learning [19, 6, 21, 17, 20, 10, 13, 12].”
Federated Semi-Supervised Learning with Inter-Client Consistency (2020, background): “FedMatch enhances this pseudo labeling process by gathering the wisdom of multiple local models as people do in a crowdsourcing environment [10].”
Artificial Intelligence and Collective Intelligence (2014, methodology): “Unfortunately, Dawid and Skene's model cannot make this inference, but Whitehill et al.'s uses information about workers' errors to update its belief about problem difficulty and hence about other workers' accuracies.”
The Multidimensional Wisdom of Crowds (2010, methodology): “We found that GLAD performs worse than majority voting when the variance in the bias between different annotators is high (γ & 0.8); this was expected as GLAD does not model annotator bias.”
Counting in the Wild (2016, unknown): “Dealing with multiple annotators has been generally approached by modelling different annotation variables with the objective of scoring and weighting the influence of each of the annotators [12, 22, 23], and finding the ground-truth label that is assumed to exist in the consensus of the annotators ”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, methodology): “Much more complex annotation aggregation methods such as probabilistic models of annotation, accounting item level effects, or clustering methods have been introduced in the recent years (Passonneau and Carpenter, 2014; Whitehill et al., 2009; Luther et al., 2015).”
Efficient Online Scalar Annotation with Bounded Support (2018, background): “” (5)There has been a line of work on relative weighting of annotators, based on their agreement with others (Whitehill et al., 2009; Welinder et al., 2010; Hovy et al., 2013).”
Crowdsourcing Translation: Professional Quality from Non-Professionals (2011, background): “Whitehill et al. (2009) proposed a probabilistic model to ﬁlter labels from non-experts, in the context of an image labeling task.”
A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality (2016, methodology): “Other methods directly estimate worker quality to improve these external checks [19, 56].”
Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “…et al. (2012); Aydin et al. (2014) use worker probability to model the ability of an annotator to correctly answer a task, and some other works (Whitehill et al., 2009; Li et al., 2014b) introduce a similar concept, worker quality, by changing the value range from [0 , 1] to ( −∞ , + ∞ ) .”