Confident Learning: Estimating Uncertainty in Dataset Labels

By Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang. 2019

1,003 citations recorded by the source corpus.

Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. Whereas numerous studies have developed these principles independently, here, we combine them, building on the assumption of a class-conditional noise process to directly estimate the joint distribution between noisy (given) labels and uncorrupted (unknown) labels. This results in a generalized CL which is provably consistent and experimentally performant. We present sufficient conditions where CL exactly finds label errors, and show CL performance exceeding seven recent competitive approaches for learning with noisy labels on the CIFAR dataset. Uniquely, the CL framework is not coupled to a specific data modality or model (e.g., we use CL to find several label errors in the presumed error-free MNIST dataset and improve sentiment classification on text data in Amazon Reviews). We also employ CL on ImageNet to quantify ontological class overlap (e.g., estimating 645 missile images are mislabeled as their parent class projectile), and moderately increase model accuracy (e.g., for ResNet) by cleaning data prior to training. These results are replicable using the open-source cleanlab release.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Are we done with ImageNet? (2020, background): “Several studies have revisited common computer vision benchmarks [16, 17, 18, 19] and, regarding ImageNet [1] specifically, identified various sources of bias and noise [20, 21, 22].”
FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, background): “10 Similarly, ImageNet data are often presumed to have correct labels, but it has been estimated that at least 100k images could be labeled incorrectly [85].”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “In fact, we notice that prior constructed datasets have experienced multiple rounds of filtering across time [88].”
Addressing Missing Labels in Large-Scale Sound Event Recognition Using a Teacher-Student Framework With Loss Masking (2020, methodology): “2 illustrate the impact of missing labels by plotting performance (d’ and lωlrap) as a function of the amount of topscored negatives discarded, similar to the treatment of noisy ImageNet labels in [26].”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “Recent methods have shown successes in overcoming training data bias such as label noise [3, 17, 25, 26, 43, 54, 57, 77] and biased data distributions [9, 11, 38, 41, 61, 69, 71].”
From ImageNet to Image Classification: Contextualizing Progress on Benchmarks (2020, background): “Still, despite our awareness of these issues (Russakovsky et al., 2015; Recht et al., 2019; Stock & Cisse, 2018; Hooker et al., 2019; Northcutt et al., 2019), we lack a precise characterization of their pervasiveness and impact, even for widely-used datasets.”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, neutral): “[340] (CL: Confident learning), Manders et al.”
Data-Centric AI (2024, positive): “For example, the authors of Northcutt et al. (2021) introduce some DCAI strategies such as the use Conﬁdent Learning to address the challenges of label quality.”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, methodology): “[31] have demonstrated that CL more accurately identifies label errors than other label110 error identification methods.”
Advances, challenges and opportunities in creating data for trustworthy AI (2022, methodology): “By looking at data points whose human annotation systematically deviates from predictions made by AI models, the Confidence Learning algorithm identified over 3% of the test data in common benchmarks such as ImageNet that are misannotated 53,54 .”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, background): “Many approaches use signals from training as a proxy for label quality [19, 46, 54, 56, 62, 64, 65].”
Overview and Importance of Data Quality for Machine Learning Tasks (2020, background): “Label Noise:Most of the large data generated or annotated have some noisy labels [14].”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “Other works such as Northcutt et al. (2021) tackles the case of non-uniform noise by estimating the noise matrix.”
Challenges in Deploying Machine Learning: A Survey of Case Studies (2020, methodology): “Some of the most popular tools include Snorkel [13], Snuba [141] and cleanlab [96].”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, neutral): “…et al., 2019), example weighting (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016), cross-validation (Northcutt et al., 2019), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017; Li et al., 2020; Zhang et al., 2020), data augmentation…”