Identifying Mislabeled Training Data

By Carla E. Brodley, Mark A. Friedl. 1999

This paper presents a new approach to identifying and eliminating mislabeled training instances for supervised learning. The goal of this approach is to improve classification accuracies produced by learning algorithms by improving the quality of the training data. Our approach uses a set of learning algorithms to create classifiers that serve as noise filters for the training data. We evaluate single algorithm, majority vote and consensus filters on five datasets that are prone to labeling errors. Our experiments illustrate that filtering significantly improves classification accuracy for noise levels up to 30%. An analytical and empirical evaluation of the precision of our approach shows that consensus filters are conservative at throwing away good data at the expense of retaining bad data and that majority filters are better at detecting bad data at the expense of throwing away good data. This suggests that for situations in which there is a paucity of data, consensus filters are preferable, whereas majority vote filters are preferable for situations with an abundance of data.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “Measurement error in Y is sometimes called noisy labels [127], mislabeled data [17, 16], or unreliable oracles [154].”
Learning kernel logistic regression in the presence of class label noise (2014, background): “The seemingly straightforward approach is by means of data preprocessing where any suspect samples are removed or relabelled [7,1,29,37,31,18].”
On Robustness of Neural Architecture Search Under Label Noise (2020, background): “The first group comprises mostly label-cleansing methods that aim to correct mislabeled data (Brodley and Friedl, 1999) or adjust the sampling weights of unreliable training instances (Chang et al.”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, background): “Second, we aim to characterize whether forgetting statistics can be used to identify “important” samples and detect outliers and examples with noisy labels (John, 1995; Brodley & Friedl, 1999; Sukhbaatar et al., 2014; Jiang et al., 2018).”
Training Deep Neural Networks on Noisy Labels with Bootstrapping (2014, background): “Brodley & Friedl (1999) developed statistical meth-ods for identifying mislabeled training data.”
Towards Making Systems Forget with Machine Unlearning (2015, neutral): “I. INTRODUCTION”
Learning from Noisy Large-Scale Datasets with Minimal Supervision (2017, background): “Approaches to learn from noisy labeled data can generally be categorized into two groups: Approaches in the ﬁrst group aim to directly learn from noisy labels and focus mainly on noise-robust algorithms, e.g., [3, 20], and label cleansing methods to remove or correct mislabeled data, e.g., [4].”
Learning from Noisy Labels with Deep Neural Networks (2014, background): “A simple approach to handle noisy labels is a data preprocessing stage, where labels suspected to be incorrect are removed or corrected [1, 3].”
Training deep neural-networks using a noise adaptation layer (2016, methodology): “A very common approach to noisy datasets is to remove the suspect samples in a preprocessing stage or have them relabeled by a data expert (Brodley & Friedl (1999)).”
Decoupling "when to update" from "how to update" (2017, methodology): “From the family of preprocessing methods, we mention [4, 10], that try to eliminate instances that are suspected to be mislabeled.”
DeepCham: Collaborative Edge-Mediated Adaptive Deep Learning for Mobile Object Recognition (2016, background): “When multiple users are involved in labeling, there is also a labelconsensus problem [9] for labeling the same object class.”
A brief introduction to weakly supervised learning (2018, background): “In practice, a basic idea is to identify the potentially mislabeled examples [13], and then try to make some correction.”
Global-Local GCN: Large-Scale Label Noise Cleansing for Face Recognition (2020, background): “Label noise cleansing is a long-term issue in building up a dataset [9, 10, 24, 25, 30, 32].”
Structured labeling for facilitating concept evolution in machine learning (2014, background): “While labeling data is a seemingly simple task, it is actually fraught with problems (e.g., [9, 19, 26]).”
Classification in the Presence of Label Noise: A Survey (2014, methodology): “To solve this problem, ensembles of classiﬁers are used in [46], [138], and [180] to identify mislabeled instances, what is inspired by outlier removal in regression [181].”
Training deep neural-networks based on unreliable labels (2016, background): “A very common approach with noisy datasets is to remove the suspect samples in a preprocessing stage or have them relabeled by a data expert [5].”
An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics (2013, background): “Noisy data is known to affect the way any data mining system behaves [20,109,151].”
Noise Tolerance Under Risk Minimization (2011, background): “Outliers detection [4], restoration of clean labels for noisy points [5], and restricting the effects of noisy points on the classifier [6], [7] are some of the well-known tricks to tackle the label noise.”