Noise Tolerance Under Risk Minimization

By Naresh Manwani, S. M. I. P. S. Sastry. 2011

290 citations recorded by the source corpus.

In this paper, we explore noise-tolerant learning of classifiers. We formulate the problem as follows. We assume that there is an unobservable training set that is noise free. The actual training set given to the learning algorithm is obtained from this ideal data set by corrupting the class label of each example. The probability that the class label of an example is corrupted is a function of the feature vector of the example. This would account for most kinds of noisy data one encounters in practice. We say that a learning method is noise tolerant if the classifiers learnt with noise-free data and with noisy data, both have the same classification accuracy on the noise-free data. In this paper, we analyze the noise-tolerance properties of risk minimization (under different loss functions). We show that risk minimization under 0-1 loss function has impressive noise-tolerance properties and that under squared error loss is tolerant only to uniform noise; risk minimization under other loss functions is not noise tolerant. We conclude this paper with some discussion on the implications of these theoretical results.

Subfields: Artificial Intelligence, Control and Systems Engineering, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Evaluation & Benchmarks

Citations in the corpus

Towards the Science of Security and Privacy in Machine Learning (2016, background): “This scenario corresponds to a poisoning attack [25], and is an instance of learning in the presence of non-necessarily adversarial but nevertheless noisy data [41].”
Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels (2015, background): “Much of the work on learning from noisy labels focuses on robust algorithms [19, 32], voting methods [2], or statistical queries [22].”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “Here we deﬁne the robustness of risk minimization algorithms (Manwani and Sastry, 2013).”
SoK: Security and Privacy in Machine Learning (2018, background): “This scenario corresponds to a poisoning attack [19], and is an instance of learning in the presence of non-necessarily adversarial but nevertheless noisy data [34].”
Learning from Noisy Large-Scale Datasets with Minimal Supervision (2017, background): “, [3, 15, 21], and label cleansing methods to remove or correct mislabeled data, e.”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “Then, the empirical risk minimization under loss function L is defined to be noise tolerant [25] if f∗ is a global minimum of the noisy risk R L(f).”
Deep Learning is Robust to Massive Label Noise (2017, background): “Approaches to learn from noisy data can generally be categorized into two groups: In the ﬁrst group, approaches aim to learn directly from noisy labels and focus on noise-robust algorithms, e.g., [1, 5, 7, 12, 13, 19].”
Learning from massive noisy labeled data for image classification (2015, methodology): “To better handle label noise, some approaches rely on training classifiers with label noise-robust algorithms [4, 15].”
Ieee Transactions on Pattern Analysis and Machine Intelligence Classification with Noisy Labels by Importance Reweighting (unknown): “Manwani and Sastry [9] analyzed the tolerance properties of RCN for risk minimization under several frequently used surrogate loss functions and showed that many of them do not tolerate RCN. Natarajan et al. [10] reported two methods for learning asymmetric RCN models, in which the random label nois”
L_DMI: An Information-theoretic Noise-robust Loss Function (2019, unknown): “, 0-1 loss [22], ramp loss [3], unhinged loss [40], savage loss [23]) have been proved to be robust to uniform or symmetric noise and Natarajan et al.”
Part-dependent Label Noise: Towards Instance-dependent Label Noise (2020, methodology): “, the random classification noise (RCN) model [8, 44, 42], the class-conditional label noise (CCN) model [50, 67, 79],”
Loss factorization, weakly supervised learning and label noise robustness (2016, background): “Proof The proof draws ideas from [Manwani and Sastry, 2013].”
Robust Loss Functions for Learning Multi-class Classifiers (2018, background): “It has been theoretically shown that risk minimiza-687 tion (in the 2-class case) is robust to symmetric or uniform label noise if the loss function satisﬁes simple conditions [2].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, unknown): “, exponential loss, log-loss, squared loss, and hingeloss) [41], [42].”
A Light CNN for Deep Face Representation With Noisy Labels (2015, background): “In the first category, robust loss [1, 23] is designed for classification tasks, so that the learnt classification models are robust to the presence of label noise.”
Classification in the Presence of Label Noise: A Survey (2014, background): “In [82], risk minimization under a given loss function is deﬁned as label noise-robust if the probability of misclassiﬁca-tion of inferred models is identical, irrespective of label noise presence.”
Robust Loss Functions under Label Noise for Deep Neural Networks (2017, background): “Risk minimization under loss function L, is said to be noise-tolerant if (Manwani and Sastry 2013) PrD[pred ◦ f∗(x) = yx] = PrD[pred ◦ f∗ η (x) = yx] Risk minimization under a given loss function is noise tolerant if the f∗ η has the same probability of misclassification as that of f∗ on the noise f”