Normalized Loss Functions for Deep Learning with Noisy Labels

By Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Monazam Erfani, James Bailey. International Conference on Machine Learning, 2020

563 citations recorded by the source corpus.

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/07cca761749bfe21c2d096ff60f32b574d5c84c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (2023, methodology): “Common methods include boot-strapping (Reed et al., 2014; Han et al., 2018; Li et al., 2020), noise-robust losses (Zhang & Sabuncu, 2018; Hendrycks et al., 2018; Ma et al., 2020), and noise modeling (Yi & Wu, 2019).”
Robust early-learning: Hindering the memorization of noisy labels (2021, methodology): “…et al., 2021), using graph models (Xiao et al., 2015; Li et al., 2017b), using surrogate loss functions (Zhang & Sabuncu, 2018; Wang et al., 2019; Ma et al., 2020), meta-learning (Ren et al., 2018; Shu et al., 2020), and employing the small loss trick (Jiang et al., 2018; Han et al., 2018b; Yu…”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, unknown): “SFAD [168] further develops a normalized symmetric cross-entropy variant by incorporating the normalized cross-entropy loss [229], and CATTAn [230] exploits the negative log-likelihood ratio between correct and competing classes [231].”
Understanding and Improving Early Stopping for Learning with Noisy Labels (2021, background): “Learning with noisy data has been well studied [18, 7, 23, 29, 22].”
Robust Multi-View Clustering With Incomplete Information (2022, methodology): “In recent years, a series of works [27], [28], [29], [30], [31] have been conducted to endow neural networks with the robustness against noisy labels.”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, unknown): “…analysis or noise synthesizing in experiments, follow the class-conditional noise (CCN) assumption (Scott, Blanchard, and Handy 2013; Zhang and Sabuncu 2018; Menon et al. 2020; Ma et al. 2020), where the label noise is in dependent of its input features conditional on the latent true label.”
Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression (2021, background): “However, there is little work on the robustness of localization losses to noisy bboxes, even though a number of methods have been proposed for robust learning with noisy labels, anchors, and bboxes [27, 26, 12, 10, 15, 37, 38, 25, 19, 20].”