Learning with Annotation Noise

By Eyal Beigman, Beata Beigman Klebanov. 2009

112 citations recorded by the source corpus.

It is usually assumed that the kind of noise existing in annotated data is random classification noise. Yet there is evidence that differences between annotators are not always random attention slips but could result from different biases towards the classification categories, at least for the harder-to-decide cases. Under an annotation generation model that takes this into account, there is a hazard that some of the training instances are actually hard cases with unreliable annotations. We show that these are relatively unproblematic for an algorithm operating under the 0--1 loss model, whereas for the commonly used voted perceptron algorithm, hard training cases could result in incorrect prediction on the uncontroversial cases at test time.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models (2019, methodology): “The bias in datasets makes its way into the models trained on them which has been analysed by various works like [Stock and Cisse, 2018, Beigman and Klebanov, 2009, Sweeney, 2013, Flores et al., 2016, Buolamwini and Gebru, 2018, Bolukbasi et al., 2016], show how the blatant gender bias in datasets…”
Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016, background): “Biases in the curation of machine learning data-sets have explored in [37, 4].”
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels (2015, background): “Much of the work on learning from noisy labels focuses on robust algorithms [19, 32], voting methods [2], or statistical queries [22].”
Learning from Noisy Large-Scale Datasets with Minimal Supervision (2017, background): “, [3, 15, 21], and label cleansing methods to remove or correct mislabeled data, e.”
Classification in the Presence of Label Noise: A Survey (2014, background): “It is demonstrated that the 0-1 loss is label noise-robust for uniform label noise [140] or when it is possible to achieve zero error rate [81]; see e.g. [74] for a discussion in the case of NNAR label noise.”
Deep Learning is Robust to Massive Label Noise (2017, background): “Approaches to learn from noisy data can generally be categorized into two groups: In the ﬁrst group, approaches aim to learn directly from noisy labels and focus on noise-robust algorithms, e.g., [1, 5, 7, 12, 13, 19].”
A Light CNN for Deep Face Representation With Noisy Labels (2015, background): “In the first category, robust loss [1, 23] is designed for classification tasks, so that the learnt classification models are robust to the presence of label noise.”
Improved Natural Language Generation via Loss Truncation (2020, background): “Third, while Gigaword dataset is known to have noise, this matches the behavior of existing annotation errors (Beigman and Klebanov, 2009; Klebanov and Beigman, 2010) and uncertainty (Kry´sci´nski et al., 2019).”
Training deep neural-networks using a noise adaptation layer (2016, methodology): “Variants that are noise robust have been proposed for the most common classifiers such as logisticregression and SVM (Frénay & Verleysen (2014), Jakramate & Kabán (2012), Beigman & Klebanov (2009)).”
Learning from massive noisy labeled data for image classification (2015, methodology): “To better handle label noise, some approaches rely on training classifiers with label noise-robust algorithms [4, 15].”
Fidelity-Weighted Learning (2017, background): “…or noisy labels can be leveraged by employing a particular architecture or deﬁning a proper loss function to avoid over-ﬁtting to imperfections of the training data (Dehghani et al., 2017b; Vahdat, 2017; Patrini et al., 2017; Beigman & Klebanov, 2009; Zeng et al., 2015; Bunescu & Mooney, 2007).”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “Extensive studies have investigated training models on noisy datasets, e.g. (Beigman and Klebanov, 2009; Natarajan et al., 2013; Brodley and Friedl, 1999) as well as some noise-estimation approaches discussed above.”
Automated Essay Scoring in the Presence of Biased Ratings (2018, result): “In this respect, our work is more comparable to (Klebanov and Beigman, 2009; Beigman and Klebanov, 2009), where authors discussed and investigated the problem of learning in the presence of biased annotators.”