Learning From Crowds

By Vikas Chandrakant Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo, Charles Florin, Luca Bogoni, Linda Moy. 2010

1,543 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Learning kernel logistic regression in the presence of class label noise (2014, methodology): “The method in [36], developed in the data space (or equivalently linear kernel case), proposes an EM algorithm to learn a similar model where the true label is modelled as a hidden variable.”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, background): “[32] proposed a two-coin model to handle noisy labels from multiple annotators.”
Annotating Twitter Data from Vulnerable Populations: Evaluating Disagreement Between Domain Experts and Graduate Student Annotators (2018, background): “Most of these efforts have focused on inter-annotator reliability, improving accuracy across annotators and reducing disagreement regarding how to interpret data [10], often without analyzing causes of disagreement [14, 18, 19, 21].”
Information verification during natural disasters (2013, background): “research on crowdsourced opinion aggregation [10, 2, 11] and voting mechanisms [1].”
A brief introduction to weakly supervised learning (2018, methodology): “For this purpose, some approaches try to construct probabilistic models and then adopt the EM algorithm for the estimation [62, 86].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, methodology): “A simple approach to tackling this scenario is to use expectationmaximization (EM)-based methods such as [144], [145] to estimate the true labels and then proceed to train the deep”
Artificial Intelligence and Collective Intelligence (2014, background): “(Raykar et al. 2010) propose a model that can not only learn about worker abilities and infer correct answers, but also jointly learn a logistic regression classifier that predicts future crowd responses or the answer — obviating the need to consult human workers in the future.”
ConStance: Modeling Annotation Contexts to Improve Stance Classification (2017, methodology): “To update the classiﬁer M , following the guidance of Raykar et al. (2010), we retrain the classiﬁer using the current estimates of Y i as weights for items.”
How does Disagreement Help Generalization against Label Corruption? (2019, background): “In the probabilistic modeling category, [28] proposed a two-coin model to handle noisy labels from multiple annotators.”
Label-Noise Robust Logistic Regression and Its Applications (2012, methodology): “There are already several works employing latent variable models of this kind, especially in the field of epidemiology, econometrics and computer-aided diagnosis (see [20, 7, 23]”
Towards interactive Machine Learning (iML): Applying Ant Colony Algorithms to Solve the Traveling Salesman Problem with the Human-in-the-Loop Approach (2016, neutral): “In the future it is planned to use gamification [45] and crowdsourcing [46], with the grand goal to make use of many humans to help to solve hard problems.”
Variational Inference for Crowdsourcing (2012, background): “3(b)-(c), both the symmetric and the two-coin versions of BP and AMF(EM) performed equally well, all achieving almost the same performance as the SpEM algorithm reported in [4].”
Error Rate Analysis of Labeling by Crowdsourcing (2013, methodology): “, 2010) or using prior over worker confusion matrices (Raykar et al., 2010; Liu et al., 2012).”
Flock: Hybrid Crowd-Machine Learning Classifiers (2015, methodology): “Our analysis of the W IKIPEDIA dataset shows promise in trying to learn classiﬁers that predicts crowd responses [37], assuming that crowd behavior can be effectively encoded in the classiﬁer’s feature space.”