How does Disagreement Help Generalization against Label Corruption?

By Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Wai-Hung Tsang, Masashi Sugiyama. 2019

972 citations recorded by the source corpus.

Learning with noisy labels is one of the hottest problems in weakly-supervised learning. Based on memorization effects of deep neural networks, training on small-loss instances becomes very promising for handling noisy labels. This fosters the state-of-the-art approach "Co-teaching" that cross-trains two deep neural networks using the small-loss trick. However, with the increase of epochs, two networks converge to a consensus and Co-teaching reduces to the self-training MentorNet. To tackle this issue, we propose a robust learning paradigm called Co-teaching+, which bridges the "Update by Disagreement" strategy with the original Co-teaching. First, two networks feed forward and predict all data, but keep prediction disagreement data only. Then, among such disagreement data, each network selects its small-loss data, but back propagates the small-loss data from its peer network and updates its own parameters. Empirical results on benchmark datasets demonstrate that Co-teaching+ is much superior to many state-of-the-art methods in the robustness of trained models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…methods that aim to correct mislabeled data (Brodley and Friedl, 1999) or adjust the sampling weights of unreliable training instances ( Chang et al., 2017; Han et al., 2018; Jiang et al., 2018; Ren et al., 2018; Yu et al., 2019) (adding Co-teaching from Han et al., 2018 and Yu et al., 2019).”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, methodology): “In modern approaches of handling noisy labels [14, 33, 38], only the clean set is used to train the model in a supervised manner, while the noisy set is discarded for robust learning.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Small loss sample selection [36, 72, 45] techniques by co-teaching [85, 21, 26, 90, 55, 11] identify noisy samples with multiple models in the same vein.”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, methodology): “Compared with Co-teaching, Co-teaching+ further employs the disagreement strategy of Decouple.”
DualGraph: A graph-based method for reasoning about label noise (2021, methodology): “These datasets are popularly used for the evaluation of learning with noisy labels in previous literatures [7, 11, 29, 36, 35].”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “g one network accumulates the noise related error whereas two networks ﬁlters noise error more successfully. Idea of co-teaching is further improved by iterating over data where two networks disagree [78], to prevent two network converging each other with increasing number of epochs. Another work us”
Denoising Implicit Feedback for Recommendation (2020, background): “Similar conclusions about noisy samples are also found in the computer vision domain [7, 12, 38], further demonstrating that the phenomenon that noisy samples are harder to fit in the early training stages isn’t model-specific, data-specific, or even domain-specific.”
A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels (2019, background): “Training deep neural networks in the presence of label noise is an active research area [21][11][38][8].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, background): “Co-teaching was further improved in [154], where the authors suggested to focus the training on data samples with lower loss values”
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights (2020, methodology): “Han et al. (2018); Yu et al. (2019) employ an approach where two networks select training examples for each other using the small-loss trick.”
From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation (2021, methodology): “Sample selection based approaches [22, 17, 14, 41] attempt to select possibly clean samples in each iteration for training.”
Denoised Maximum Classifier Discrepancy for Source-Free Unsupervised Domain Adaptation (2022, methodology): “Based on this observation, some approaches (Han et al. 2018; Yu et al. 2019) have achieved good results by filtering labels to reduce the accumulation of errors.”
Curriculum Loss: Robust Learning and Generalization against Label Corruption (2019, methodology): “For Co-teaching+ (Yu et al., 2019), we employ the open-sourced code on github.”
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee (2019, background): “, 2017), and training two networks simultaneously (Malach and Shalev-Shwartz, 2017; Han et al., 2018; Yu et al., 2019).”
DivideMix: Learning with Noisy Labels as Semi-supervised Learning (2020, methodology): “Compared to Co-teaching (Han et al., 2018) and Co-teaching + (Yu et al., 2019), our method is more robust to noise by enabling the two networks to teach each other implicitly at each epoch (co-divide) and explicitly at each mini-batch (label co-reﬁnement and co-guessing). its per-sample loss…”