Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels

By Pengfei Chen, Ben Liao, Guangyong Chen, Shengyu Zhang. 2019

460 citations recorded by the source corpus.

Noisy labels are ubiquitous in real-world datasets, which poses a challenge for robustly training deep neural networks (DNNs) as DNNs usually have the high capacity to memorize the noisy labels. In this paper, we find that the test accuracy can be quantitatively characterized in terms of the noise ratio in datasets. In particular, the test accuracy is a quadratic function of the noise ratio in the case of symmetric noise, which explains the experimental findings previously published. Based on our analysis, we apply cross-validation to randomly split noisy datasets, which identifies most samples that have correct labels. Then we adopt the Co-teaching strategy which takes full advantage of the identified samples to train DNNs robustly against noisy labels. Compared with extensive state-of-the-art methods, our strategy consistently improves the generalization performance of DNNs under both synthetic and real-world training noise.

Subfields: Artificial Intelligence, Civil and Structural Engineering, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Civil and Structural Engineering

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “Table 3 compares the purification performance with the state-of-the-art noise detection methods based on the training dynamics, including AUM [64] and INCV [11].”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “, 2018), cross validation (Chen et al., 2019), Shapely values (Ghorbani and Zou, 2019), and the area-under-margin metric (Pleiss et al.”
MoPro: Webly Supervised Learning with Momentum Prototypes (2020, methodology): “, 2018; 2020), iterative training (Tanaka et al., 2018; Yi & Wu, 2019; Wang et al., 2018; Chen et al., 2019), or meta-learning (Li et al.”
Confident Learning: Estimating Uncertainty in Dataset Labels (2019, methodology): “This is identical to using the off-diagonals of Cconfusion and similar to a single iteration of INCV (Chen et al., 2019).”
DivideMix: Learning with Noisy Labels as Semi-supervised Learning (2020, methodology): “Cross-Entropy 85.0 72.3 F-correction (Patrini et al., 2017) 87.2 83.1 M-correction (Arazo et al., 2019) 87.4 86.3 Iterative-CV (Chen et al., 2019) 88.6 88.0 P-correction (Yi & Wu, 2019) 88.5 88.1 Joint-Optim (Tanaka et al., 2018) 88.9 88.4 Meta-Learning (Li et al., 2019) 89.2 88.6 DivideMix 93.4 92.”
Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, methodology): “To achieve this, we lean on a number of contributions in 104 uncertainty quantification for finding label errors based on prediction/label agreement via confusion 105 matrices [3, 15, 24, 45], however these approaches lack either robustness to class imbalance or 106 (4)A corrected version of each te”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…exist e.g. image corruption noise (Hendrycks & Dietterich, 2019), image registration noise (Mnih & Hinton, 2012), or noise from adversarial attacks (Zhang et al., 2019), this paper focuses on label noise, and in particular web label noise – a common type of label noise used in the literature.”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, methodology): “Prior methods deal with mislabeled data through various methods, such as validation loss on rotating hold-out sets (Chen et al., 2019) or robust loss functions (Reed et al.”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, methodology): “F-correction [26] 38.88 (17.32) 42.64 (17.64) Decouple [25] 37.46 (15.26) 41.74 (17.74) Co-teaching [9] 36.42 (14.80) 38.52 (15.30) MentorNet [12] 37.00 (18.60) 42.20 (20.08) D2L [23] 37.32 (16.00) 42.20 (18.64) INCV [5] 34.76 (14.66) 38.40 (15.02) MORPH 29.98 (11.33) 33.09 (12.62) Table 2: Comparis”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “uncertain samples [71], extra layer with similarity matrix [72], curriculum loss [73], data complexity [74], partial labels [75] c.Multiple Classiﬁers: Consistency of networks [76], co-teaching [77]– [79] d.Active Learning: Relabel hard samples for classiﬁer [39] 4.Sample Importance Weighting Estima”
Early-Learning Regularization Prevents Memorization of Noisy Labels (2020, methodology): “For ease of comparison to previous works [17, 7], we consider the mini WebVision dataset which contains the top 50 classes from the Google image subset of WebVision, which results in approximately 66 thousand images.”