Learning From Noisy Singly-labeled Data

By Ashish Khetan, Zachary Chase Lipton, Anima Anandkumar. 2017

170 citations recorded by the source corpus.

Supervised learning depends on annotated examples, which are taken to be the \emph{ground truth}. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality is above a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “Another group of approaches treats the true but unknown labels as latent variables and the noisy labels as observed variables so that EM-like algorithms can be used to learn the true label distribution of the dataset (Xiao et al., 2015; Vahdat, 2017; Khetan et al., 2018).”
A Survey on Programmatic Weak Supervision (2022, methodology): “Crowdsourcing models [Dawid and Skene, 1979; Dalvi et al. , 2013; Raykar et al. , 2010; Khetan et al. , 2018] usually leverage the expectation maximization (EM) algo-rithm to estimate the accuracy of each worker as well as infer the latent ground truth labels, which can also be applied here when we…”
Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training (2019, methodology): “Our approach is inspired in the Model Bootstrapped Expectation Maximization (MBEM) algorithm of Khetan et al. (2018).”
Tilted Empirical Risk Minimization (2020, methodology): “Significant research has thus been devoted to developing alternatives to traditional ERM for diverse applications, such as learning in the presence of noisy/corrupted data (Jiang et al., 2018; Khetan et al., 2018), performing classification with imbalanced data (Lin et al.”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “While a number of relevant works address the setting where annotator labels are available (Bouguelia et al., 2018; Tanno et al., 2019a,b; Khetan et al., 2018), this paper addresses the general setting where no annotation information is available except the observed noisy labels.”
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options (2020, background): “When the annotation budget is fixed, Khetan et al. (2018) suggest that it is better to label collect single label per training example as many as possible, rather than collecting less training examples with multiple labels.”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, methodology): “Data reweighting [87] - image classification lesion detection ([88]) and segmentation (experiments in Section V-A), lesion classification ([89]; experiments in Section V-B), segmentation ([90]; [91]) [92] - image classification [93] - image classification [94] - image classification [95] - image cla”
Training Complex Models with Multi-Task Weak Supervision (2018, methodology): “Our work builds on and extends various settings studied in machine learning: Weak Supervision: We draw motivation from recent work which models and integrates weak supervision using generative models [30; 29; 2] and other methods [13; 19].”
NLNL: Negative Learning for Noisy Labels (2019, unknown): “Other approaches include jointly modeling labels and worker quality [15], creating a robust method to learn in open-set noisy label situations [34] and attempting to prune the correct samples [6, 2, 24].”
Learning from Rules Generalizing Labeled Exemplars (2020, background): “Learning with noisy labels with or without additional clean data has been a problem of long-standing interest in ML (Khetan et al., 2018; Zhang & Sabuncu, 2018; Ren et al., 2018b; Veit et al., 2017; Shen & Sanghavi, 2019).”
Curriculum Loss: Robust Learning and Generalization against Label Corruption (2019, background): “For instance, noisy corrupted (wrong) labels may be resulted from annotating for similar objects (Su et al., 2012; Yuan et al., 2019), crawling images and labels from websites (Hu et al., 2017; Tanaka et al., 2018) and creating training sets by program (Ratner et al., 2016; Khetan et al., 2018).”
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks (2019, background): “In addition to early stopping, several heuristics have been proposed to speciﬁcally deal with label noise [26,30,36,42,47,57].”
Learning with Bad Training Data via Iterative Trimmed Loss Minimization (2018, background): “On the other hand, (Khetan et al., 2018) and (Zhang & Sabuncu, 2018) provide new losses for training under the noise.”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “EM [104, 105, 26], trace regularizer [106], crowd-layer [107], image difficulty estimate [108] consistency with network [109], omitting probability variable [110] softmax layer per labeler [25]”
What is the Effect of Importance Weighting in Deep Learning? (2018, methodology): “…function of the input x : q ( x ) /p ( x ) (in the case of covariate shift) or of the label y : q ( y ) /p ( y ) (in the case of label shift), training a corrected classiﬁer with importance-weighted empirical risk minimization (IW-ERM) [Shimodaira, 2000, Gretton et al., 2009, Lipton et al., 2018].”
Weakly supervised classification of rare aortic valve malformations using unlabeled cardiac MRI sequences (2018, background): “Noise is commonly addressed using rule-based and generative modeling strategies for estimating the accuracy of label sources [24,25].”