Loss factorization, weakly supervised learning and label noise robustness

By Giorgio Patrini, Frank Nielsen, Richard Nock, M. Carioni. 2016

120 citations recorded by the source corpus.

We prove that the empirical risk of most well-known loss functions factors into a linear term aggregating all labels with a term that is label free, and can further be expressed by sums of the loss. This holds true even for non-smooth, non-convex losses and in any RKHS. The first term is a (kernel) mean operator --the focal quantity of this work-- which we characterize as the sufficient statistic for the labels. The result tightens known generalization bounds and sheds new light on their interpretation. Factorization has a direct application on weakly supervised learning. In particular, we demonstrate that algorithms like SGD and proximal methods can be adapted with minimal effort to handle weak supervision, once the mean operator has been estimated. We apply this idea to learning with asymmetric noisy labels, connecting and extending prior work. Furthermore, we show that most losses enjoy a data-dependent (by the mean operator) form of noise robustness, in contrast with known negative results.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistics and Probability

Citations in the corpus

Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption (2017, methodology): “…statistics for the class instead of examples [Nock et al., 2015, Nock, 2016, Patrini, 2016], many of which do not require entity resolution to be computed, but rather a weak form of entity resolution between groups of observations that share common subset of features [Patrini et al., 2016b].”
Robust Loss Functions for Learning Multi-class Classifiers (2018, result): “Patrini et al. [17] provide similar results in semi-supervised learning for linear odd losses, improving the bounds of [16].”
Does label smoothing mitigate label noise? (2020, background): “Indeed, label smoothing is conspicuously absent in most treatments of the noisy label problem (Patrini et al., 2016; Han et al., 2018b; Charoenphakdee et al., 2019; Thulasidasan et al., 2019; Amid et al., 2019; Menon et al., 2020).”
Decoupling "when to update" from "how to update" (2017, background): “The effects of noisy labels was vastly studied in many different learning algorithms (see for example the survey in [13]), and various solutions to this problem have been proposed, some of them with theoretically provable bounds, including methods like statistical queries, boosting, bagging and more”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, unknown): “The CCN assumption is commonly used in previous works, as clearly stated in theoretical analysis (Blum and Mitchell 1998; Yan et al. 2017; Patrini et al. 2016; Zhang and Sabuncu 2018; Xu et al. 2019; Menon et al. 2020; Ma et al. 2020) or inexplicitly used in experiments for synthetizing noisy…”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, unknown): “, exponential loss, log-loss, squared loss, and hingeloss) [41], [42].”
Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach (2016, methodology): “The proof makes use of the factorization trick due to [30].”
Positive-Unlabeled Reward Learning (2019, background): “Most existing works in PU learning require certain loss functions, linear models, and/or special optimizers (du Plessis et al., 2014; 2015b; Patrini et al., 2016).”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, background): “Experiments in learning with noisy labels [19, 31, 34, 42, 45] suffer a double-edged sword: either synthetic noise must be added to clean training data to measure performance on a clean test set (at the expense of modeling actual real-world label noise [18]), or a naturally noisy dataset is used…”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “…categories: (1) theoretical contributions (Katz-Samuels et al., 2017; Blanchard and Scott, 2014), (2) modifying the loss for label noise robustness (Patrini et al., 2016, 2017b; Sukhbaatar et al., 2014; van Rooyen et al., 2015), (3) deep learning and model-specific approaches (Sukhbaatar et al.,…”