Learning with Symmetric Label Noise: The Importance of Being Unhinged

By Brendan van Rooyen, Aditya Krishna Menon, Robert C. Williamson. Neural Information Processing Systems, 2015

356 citations recorded by the source corpus.

Convex potential minimisation is the de facto approach to binary classification. However, Long and Servedio [2010] proved that under symmetric label noise (SLN), minimisation of any convex potential over a linear function class can result in classification performance equivalent to random guessing. This ostensibly shows that convex losses are not SLN-robust. In this paper, we propose a convex, classification-calibrated loss and prove that it is SLN-robust. The loss avoids the Long and Servedio [2010] result by virtue of being negatively unbounded. The loss is a modification of the hinge loss, where one does not clamp at zero; hence, we call it the unhinged loss. We show that the optimal unhinged solution is equivalent to that of a strongly regularised SVM, and is the limiting solution for any convex potential; this implies that strong l2 regularisation makes most standard learners SLN-robust. Experiments confirm the unhinged loss' SLN-robustness is borne out in practice. So, with apologies to Wilde [1895], while the truth is rarely pure, it can be simple.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/dd05ce86cacf50e3aee70d633040241923deb120

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

How does Disagreement Help Generalization against Label Corruption? (2019, background): “Assume that the matrix Q has two representative structures: (1) Symmetry flipping [34]; (2) Pair flipping [9]: a simulation of fine-grained classification with noisy labels, where labelers may make mistakes only within very similar classes.”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, background): “Unlike prior work that only models symmetric label noise [43], we 111 quantify class-conditional label noise with CL, validating the correctable nature of the label errors 112 via crowdsourced workers.”
L_DMI: An Information-theoretic Noise-robust Loss Function (2019, unknown): “, 0-1 loss [22], ramp loss [3], unhinged loss [40], savage loss [23]) have been proved to be robust to uniform or symmetric noise and Natarajan et al.”
On Symmetric Losses for Learning from Corrupted Labels (2019, background): “Not limited to the simpliﬁcation of the risk estimator, symmetric losses are known to be robust in the symmetric label noise scenarios [Angluin & Laird, 1988; Ghosh et al., 2015, 2017; Long & Servedio, 2010; Manwani & Sastry, 2013; Van Rooyen et al., 2015].”
On Regularizing Rademacher Observation Losses (2016, background): “Some popular losses fit in the equivalence [Nair and Hinton, 2010, Gentile and Warmuth, 1998, Nock and Nielsen, 2008, Telgarsky, 2012, Vapnik, 1998, van Rooyen et al., 2015].”
Learning Games and Rademacher Observations Losses (2015, background): “…even when not all ofthem are systematically used as lossesp r se: exponential, logistic, square, mean-variance, ReLU, linear Hinge, and unhinged losses (Nair & Hinton, 2010; Gentile & Warmuth, 1998; Nock & Nielsen, 2008; Telgarsky, 2012; Vapnik, 1998; van Rooyen et al., 2015) (and many others).”
Robust Loss Functions under Label Noise for Deep Neural Networks (2017, background): “Unhinged loss, a convex loss (which is not convex potential), is robust to symmetric label noise [27].”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “(4) With this particular configuration of DNN, the proposed MAE loss is, up to a constant of proportionality, the same as the unhinged loss Lunh(f(x), ej) = 1− fj(x) [37].”
Transfer Learning with Label Noise (2017, background): “Learning with noisy labels in the traditional classiﬁcation problem has been widely studied [13, 24].”
Entity Resolution and Federated Learning get a Federated Resolution (2018, background): “With the exception of the unhinged loss [27], this would be the case for most proper, classification calibrated and/or nonconvex losses [3, 21, 24].”
A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels (2019, methodology): “We have used two different noise models: (1) symmetry flipping [36] (symmetry) and (2) pair flipping (pair).”
NLNL: Negative Learning for Noisy Labels (2019, methodology): “Some methods attempted to create noise-robust losses [1, 32, 23, 4, 3, 39].”
Joint Optimization Framework for Learning with Noisy Labels (2018, background): “Robust loss function: A few studies achieve noiserobust classification by using a noise-tolerant loss functions, such as ramp loss [3] and unhinged loss [20].”
Weakly supervised learning via statistical sufficiency (2016, methodology): “This simple proof can be extended and used for unhinged loss, if we assume bounded models [van Rooyen et al., 2015].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…the data during training (Reed et al. 2015), jointly optimizing the model’s parameters and estimating true labels (Tanaka et al. 2018), using noise-tolerant loss function (Ghosh, Kumar, and Sastry 2017; Van Rooyen, Menon, and Williamson 2015) or noise tolerant training algorithms (Li et al. 2019).”