Symmetric Cross Entropy for Robust Learning With Noisy Labels

By Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, James Bailey. 2019

1,167 citations recorded by the source corpus.

Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy labels on some classes (``easy" classes), but more surprisingly, it also suffers from significant under learning on some other classes (``hard" classes). Intuitively, CE requires an extra term to facilitate learning of hard classes, and more importantly, this term should be noise tolerant, so as to avoid overfitting to noisy labels. Inspired by the symmetric KL-divergence, we propose the approach of Symmetric cross entropy Learning (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). Our proposed SL approach simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels. We provide a theoretical analysis of SL and also empirically show, on a range of benchmark and real-world datasets, that SL outperforms state-of-the-art methods. We also show that SL can be easily incorporated into existing methods in order to further enhance their performance.

Subfields: Artificial Intelligence, Civil and Structural Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

Citations in the corpus

Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “Both variants outperform all the baselines (excluding L2R) in all three datasets, and combining them our model performs the best on MNIST and CIFAR-10 with all noise levels.”
Delving Deep Into Label Smoothing (2020, neutral): “Wang et al. [49] improved the robustness against noisy labels by replacing the normal cross-entropy function with the symmetric cross-entropy function.”
Early-Learning Regularization Prevents Memorization of Noisy Labels (2020, background): “…include Mean Absolute Error (MAE) [10], Generalized Cross Entropy [54], which can be interpreted as a generalization of MAE, Symmetric Cross Entropy [43], which adds a reverse cross-entropy term to the usual cross-entropy loss, and L DIM [46], which is based on information-theoretic considerations.”
Denoised Maximum Classifier Discrepancy for Source-Free Unsupervised Domain Adaptation (2022, methodology): “The regularization method involves a regularization term in the training loss to avoid overfitting on noise labels (Wang et al. 2019; M¨uller, Kornblith, and Hinton 2019).”
Learning From Noisy Labels With Deep Neural Networks: A Survey (2020, methodology): “In addition, inspired by the symmetricity of the Kullback–Leibler divergence, the symmetric cross entropy (SCE) [98] was proposed by combining a noise tolerance term, namely reverse cross entropy loss, with the standard CCE loss.”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, positive): “Inspired by a classic noisy label learning approach [228], BMD [122] and OnDA [208] employ the symmetric crossentropy loss to guide the self-labeling process.”
Understanding and Improving Early Stopping for Learning with Noisy Labels (2021, background): “Learning with noisy data has been well studied [18, 7, 23, 29, 22].”
Robust early-learning: Hindering the memorization of noisy labels (2021, methodology): “…et al., 2020a; Li et al., 2021), using graph models (Xiao et al., 2015; Li et al., 2017b), using surrogate loss functions (Zhang & Sabuncu, 2018; Wang et al., 2019; Ma et al., 2020), meta-learning (Ren et al., 2018; Shu et al., 2020), and employing the small loss trick (Jiang et al., 2018; Han…”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, methodology): “r [24], crowd-layer [95] Noise Model Free Methods 1.Robust Losses Risk minimization [96], [97], 0-1 loss surrogate [98], MAE [99], IMEA [100], Generalized cross-entropy [101], symmetric cross entropy [102], linear-odd losses [103], classiﬁcation calibrated losses [104], unbiased estimator [105], mod”
Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression (2021, background): “However, there is little work on the robustness of localization losses to noisy bboxes, even though a number of methods have been proposed for robust learning with noisy labels, anchors, and bboxes [27, 26, 12, 10, 15, 37, 38, 25, 19, 20].”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, background): “9 SCE-loss (Wang et al., 2019) 61.”
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (2021, methodology): “A straightforward way is to design robust losses [62, 71], but these methods fail to handle real-world noisy data.”
From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation (2021, background): “To reduce the negative effect of noisy labels, some approaches focus on training noise robust models by designing robust losses [12, 34, 44] or by correcting the loss with a transition matrix [27, 13, 37].”
TILDE: Term Independent Likelihood moDEl for Passage Re-ranking (2021, positive): “However, inspired by Symmetric Cross Entropy [24], an alternative way of training the full TILDE-QDLmodel is to leverage both LQL and LDL at the same time.”
Normalized Loss Functions for Deep Learning with Noisy Labels (2020, methodology): “On CIFAR-100 with 0.4 asymmetric noise, the highest accuracy that can be achieved by current methods is 42.19% (by SCE), which is still 5% lower than our NCE+MAE and 4% lower than our NCE+RCE.”
Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, methodology): “Moreover, when compared to other baselines incorporating dedicated noisy-aware learning techniques [52, 35], AttrPrompt still achieves promising performance, despite using the standard cross-entropy loss.”
SELF: Learning to Filter Noisy Labels with Self-Ensembling (2019, neutral): “, 2018), SL (Wang et al., 2019), Lq (Zhang & Sabuncu, 2018), Trunc Lq (Zhang & Sabuncu, 2018), Forward T̂ (Patrini et al.”
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations (2021, background): “…labels [Liu and Tao, 2015], propose robust loss functions [Natarajan et al., 2013, Ghosh et al., 2017, Zhang and Sabuncu, 2018, Amid et al., 2019, Wang et al., 2019, Liu and Guo, 2020], or introduce a robust regularizer [Liu et al., 2020, Xia et al., 2020a, Cheng et al., 2021, Wei et al., 2021].”