Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels

By Curtis G. Northcutt, Tailin Wu, Isaac L. Chuang. 2017

174 citations recorded by the source corpus.

Noisy PN learning is the problem of binary classification when training examples may be mislabeled (flipped) uniformly with noise rate rho1 for positive examples and rho0 for negative examples. We propose Rank Pruning (RP) to solve noisy PN learning and the open problem of estimating the noise rates, i.e. the fraction of wrong positive and negative labels. Unlike prior solutions, RP is time-efficient and general, requiring O(T) for any unrestricted choice of probabilistic classifier with T fitting time. We prove RP has consistent noise estimation and equivalent expected risk as learning with uncorrupted labels in ideal conditions, and derive closed-form solutions when conditions are non-ideal. RP achieves state-of-the-art noise estimation and F1, error, and AUC-PR for both MNIST and CIFAR datasets, regardless of the amount of noise and performs similarly impressively when a large portion of training examples are noise drawn from a third distribution. To highlight, RP with a CNN classifier can predict if an MNIST digit is a "one"or "not" with only 0.25% error, and 0.46 error across all digits, even when 50% of positive examples are mislabeled and 50% of observed positive labels are mislabeled negative examples.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, background): “On the other hand, to make models more robust to outliers, some approaches inject bias into the loss function in order to emphasize easier examples [37, 48, 27, 35].”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “Some approaches avoid hard examples altogether (Bottou et al., 2005; Northcutt et al., 2017) to reduce fit to noisy data.”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “[30] offered to prune the correct samples based on softmax outputs.”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, methodology): “This assumption is commonly used [9, 32, 42] because it is reasonable.”
Learning from positive and unlabeled data: a survey (2018, methodology): “The non-traditional classifier (Elkan and Noto 2008; Northcutt et al. 2017) and some partial matching (Du Plessis and Sugiyama 2014) approaches make the assumption that the positive and negative example are separable.”
NLNL: Negative Learning for Noisy Labels (2019, background): “Other approaches include jointly modeling labels and worker quality [15], creating a robust method to learn in open-set noisy label situations [34] and attempting to prune the correct samples [6, 2, 24].”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “In [72], with the help of a probabilistic classifier,”
Noise-tolerant fair classification (2019, methodology): “Algorithms that deal with CCN corruption have been well studied (Natarajan et al., 2013; Liu & Tao, 2016; Northcutt et al., 2017).”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, methodology): “Rank Pruning, proposed by [73], identifies data points with confident labels and updates the classifier using only those data points.”
Self-Supervised Representation Learning by Rotation Feature Decoupling (2019, methodology): “Then the mislabeled rate can be used to handle noisy observed negative examples by various ways such as excluding examples with low confidence [40], labeling examples with high confidence [49, 23, 19] or reweighting examples [14, 35, 32].”
Part-dependent Label Noise: Towards Instance-dependent Label Noise (2020, background): “For example, additional information is given [6]; the matrix is symmetric [43]; the noise rates for instances are upper bounded [9], or even to be instance-independent [67, 17, 50, 48, 44], i.”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “, 2016), error removal (Wang et al., 2018; Northcutt et al., 2017), or fixing labels (Han et al.”
Are Anchor Points Really Indispensable in Label-Noise Learning? (2019, background): “, P (Ȳ = j|Y = i,X = x) = P (Ȳ = j|Y = i), which is identifiable under mild conditions and on which the vast majority of current methods focus [14, 13, 30, 29, 26].”