SELF: Learning to Filter Noisy Labels with Self-Ensembling

By Duc Minh Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, T. Nguyen, Laura Beggel, Thomas Brox. 2019

358 citations recorded by the source corpus.

Deep neural networks (DNNs) have been shown to over-fit a dataset when being trained with noisy labels for a long enough time. To overcome this problem, we present a simple and effective method self-ensemble label filtering (SELF) to progressively filter out the wrong labels during training. Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the filtered noisy labels. For the filtering, we form running averages of predictions over the entire training dataset using the network output at different training epochs. We show that these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch. While filtered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss. We demonstrate the positive effect of such an approach on various image classification tasks under both symmetric and asymmetric label noise and at different noise ratios. It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

Subfields: Artificial Intelligence, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiawei Han
2. Tongliang Liu
3. Hwanjun Song
4. P. Heng
5. Jiaxin Huang
6. Yu Meng
7. Geoff Pleiss
8. Jonghyun Choi

In Natural Language Processing

1. Gang Niu
2. Bo Han
3. Jiaxin Huang
4. Yu Zhang
5. Yu Meng
6. Jiawei Han
7. Hwanjun Song
8. Xiaobo Xia

In Optimization & Training

1. Hwanjun Song
2. Geoff Pleiss
3. Tongliang Liu
4. Tianyi Zhang
5. Chen Gong
6. Jiawei Han
7. Yi Chang
8. Jingwei Zhao

Citations in the corpus

• Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “This also helps suppress the fluctuation in model predictions due to data noise, offering better noise-robustness [45].”
• Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, background): “Unlike SELFIE and Co-teaching, DivideMix considerably improves the accuracy in CIFAR-10, but not in CIFAR-100 with asymmetric noise and real noise datasets, which have more realistic label noise.”
• Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, positive): “However, the above (i) and (ii) typically require carefully tuning in existing methods (Reed et al. 2015; Ma et al. 2018; Tanaka et al. 2018; Song, Kim, and Lee 2019; Nguyen et al. 2020; Arazo et al. 2019).”
• Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, methodology): “…set of correctly-labeled data and then use the remaining untrusted data in conjunction with semi-supervised learning (Luo et al., 2018; Li et al., 2020; Ortego et al., 2019; Nguyen et al., 2020), pseudo-labeling (Zhang et al., 2019), or MixUp augmentation (Arazo et al., 2019; Zhang et al., 2018).”
• A Noise-Robust Framework for Automatic Segmentation of COVID-19 Pneumonia Lesions From CT Images (2020, methodology): “In [28], a self-ensemble label filtering method was proposed to progressively filter out wrong labels during training.”
• ADBench: Anomaly Detection Benchmark (2022, background): “…studies [131, 152] explored anomaly contamination in the unlabeled samples, we further discuss the more generalized impact of label contamination on the algorithm performance, where the label flips [122, 202] between the normal samples and anomalies are considered (up to 50% of total labels).”
• From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation (2021, background): “It is shown in [25] that the teacher network is able to provide more stable learning signal than the pseudo labels since it is a temporal ensemble of networks.”
• Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning (2022, background): “This not only neutralizes the fluctuation in model predictions for better training stability when label noise is present (Nguyen et al., 2020) but also helps prevent catastrophic forgetting (Kirk-patrick et al., 2017) of the information learned previously from the few-shot training set D train .…”
• Robust early-learning: Hindering the memorization of noisy labels (2021, methodology): “Note that we do not compare with some state-of-the-art methods like SELF (Nguyen et al., 2020) and DivideMix (Li et al.”
• Understanding and Improving Early Stopping for Learning with Noisy Labels (2021, methodology): “Current methods [27, 25] usually adopt an early stopping strategy, which decides the stopping point by considering the network as a whole.”
• Generalized Out-of-Distribution Detection: A Survey (2021, neutral): “3: Classification-based Methods [252], [253], [269], [289], [290], [291], [292]”
• Noise against noise: stochastic label noise helps combat inherent label noise (2021, background): “7) One can combat label noise with refined training strategies (Li et al., 2019b; 2020; Nguyen et al., 2020) that potentially incorporate several techniques, including sample selection/weighting, label correction, meta-learning (Li et al., 2019b) and semi-supervised learning (Tarvainen & Valpola,…”