Supervised learning from multiple experts: whom to trust when everyone lies a bit

By Vikas Chandrakant Raykar, Shipeng Yu, Linda H. Zhao, Anna K. Jerebko, Charles Florin, Gerardo Hermosillo, Luca Bogoni, Linda Moy. International Conference on Machine Learning, 2009

391 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications

View Paper: https://www.semanticscholar.org/paper/5b49178c0638d026ae49d9d1b89cee4575d6833d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, neutral): “EM [104, 105, 26], trace regularizer [106], crowd-layer [107], image difficulty estimate [108] consistency with network [109], omitting probability variable [110] softmax layer per labeler [25]”
Query-Oriented Data Cleaning with Oracles (2015, background): “As previously mentioned, extensive research has also been devoted to develop algorithms to ensure the quality of answers, both for individual answers (e.g., outlier detection [42, 59, 16]) and aggregated answers (e.g., using error probability, or an average weighted by trust [48, 47, 54]).”
Learning From Crowds (2010, background): “A preliminary version of this paper (Raykar et al., 2009) mainly discussed the binary classification problem.”
Crowdsourced Data Management: Industry and Academic Perspectives (2015, background): “These papers consider increasingly expressive models for this estimation problem, including worker bias [100], difficulty of tasks and worker expertise [193, 157, 106, 108], adversarial behavior [156], and online evaluation of workers [191, 128, 155].”
Boundary Detection Benchmarking: Beyond F-Measures (2013, background): “Many seminal models [18, 26] have been proposed to analyze the crowdsourced annotation process in general.”
Becoming the expert - interactive multi-class machine teaching (2015, background): “While it is possible to model the uncertainty and noise generated from groups of annotators to improve their collective performance [34, 40, 27], these approaches tend to downweight votes from weak annotators by learning to trust the experts.”
A Survey on Truth Discovery (2015, background): “However, workers’ quality are quite diverse, which brings the core task of inferring true labels from the labeling eﬀorts of multiple workers [6,10,24,36,39,40, 43, 50, 51, 59].”
Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection (2015, background): “ers who have adequate expertise for fine-grained classification [36, 38, 28, 22].”
Comprehensive and reliable crowd assessment algorithms (2014, background): “B) We present experimental results on synthetic data (Section IV-B) and 3 real datasets (Section IV-C) to study the performance of our confidence intervals on k-ary tasks.”
crowdsourcing : rating annotators and obtaining cost-effective labels (2010, neutral): “Annotator models have also been used to train classifiers with noisy labels [3].”
The Importance of Calibration for Estimating Proportions from Annotations (2018, background): “Another line of work has approached the problem of aggregating labels from multiple annotators (Raykar et al., 2009; Hovy et al., 2013; Yan et al., 2013).”
Beat the Machine: Challenging Humans to Find a Predictive Model's “Unknown Unknowns” (2015, background): “Thus, we would have to invest in labeling more than a million randomly selected web pages just to get one hate speech example, and, as has been pointed out recently, often you need more than one label per page to get high-quality labeling [Sheng et al. 2008; Raykar et al. 2009].”
Machine Learning and Decision Support in Critical Care (2016, methodology): “[122], [123] extended the algorithm to jointly model the ground truth and a regression model.”
Active Learning from Crowds (2011, background): “The idea of trying to model different levels of expertise among labelers has been addressed in (Raykar et al., 2009; Yan et al., 2010) and later in (Kasneci et al., 2011).”
The Multidimensional Wisdom of Crowds (2010, methodology): “Although approaches for modeling certain aspects of the annotation process have been proposed in the past [1, 5, 6, 9, 13, 4, 12], no attempt has been made to blend all characteristics of the process into a single unified model.”