Co-teaching: Robust training of deep neural networks with extremely noisy labels

By Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, Masashi Sugiyama. 2018

2,654 citations recorded by the source corpus.

Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Estimates relying on model training dynamics (Toneva et al., 2018; Swayamdipta et al., 2020), gradient magnitudes (Vodra-halli et al., 2018), or loss magnitudes (Han et al., 2018) are sensitive to factors such as variance during steps of training.”
On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…methods that aim to correct mislabeled data (Brodley and Friedl, 1999) or adjust the sampling weights of unreliable training instances (Chang et al., 2017; Han et al., 2018; Jiang et al., 2018; Ren et al., 2018; Yu et al., 2019) (adding Co-teaching from Han et al., 2018 and Yu et al., 2019).”
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, background): “Also note that most prior works (Goldberger & Ben-Reuven, 2017; Ma et al., 2018; Han et al., 2018) only experiment with small-scale images since label corruption demonstrations can require training hundreds of models (Hendrycks et al.”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, methodology): “Despite the use of multiple networks, Co-teaching [14] marginally improves the performance.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Small loss sample selection [36, 72, 45] techniques by co-teaching [85, 21, 26, 90, 55, 11] identify noisy samples with multiple models in the same vein.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, methodology): “Other approaches have used training loss (Han et al., 2018; Arazo et al., 2019; Shen and Sang-havi, 2019), conﬁdence (Hovy et al., 2013), and meta-learning (Ren et al., 2018), to differentiate instances within datasets.”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “The paradigm of co-teaching is introduced in [90], where two”
Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee (2019, background): “This kernel was named the neural tangent kernel (NTK) by Jacot et al. (2018). Although this kernel is random, it is shown that when the network is sufficiently wide, this random kernel converges to a deterministic limit in probability (Arora et al.”
NLNL: Negative Learning for Noisy Labels (2019, methodology): “The existing pruning and label cleaning methods [6, 2, 26, 30, 22] use a network trained directly with a given noisy label; thus overfitting to a noisy label can occur even if the pruning or cleaning process is performed.”
Normalized Loss Functions for Deep Learning with Noisy Labels (2020, background): “This approach improves robustness by modifying the loss function during training, based on label-dependent weights (Natarajan et al., 2013) or an estimated noise transition matrix that deﬁnes the probability of mislabeling one class with another (Han et al., 2018a).”
Robust Multi-View Clustering With Incomplete Information (2022, background): “In recent years, a series of works [27], [28], [29], [30], [31] have been conducted to endow neural networks with the robustness against noisy labels.”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, methodology): “Both are popularly used for evaluation of noisy labels [59, 16].”
SELF: Learning to Filter Noisy Labels with Self-Ensembling (2019, methodology): “We compare our work to previous methods from Reed-Hard (Reed et al., 2014), S-model (Gold-berger & Ben-Reuven, 2016), Wang et al. (2018), Rand. weights (Ren et al., 2018), Bi-level-model (Jenni & Favaro, 2018), MentorNet (Jiang et al., 2017), L q (Zhang & Sabuncu, 2018), Trunc L q (Zhang & Sabuncu, ”
A Survey on Deep Semi-Supervised Learning (2021, background): “[125] deals with this problem from the perspective of memorization effects, which”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…noise), in which the label of each example is independently and uniformly changed to a random (incorrect) class with a probability (Zhang et al., 2017; Arpit et al., 2017; Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018; Li et al., 2019; Arazo et al., 2019).”
Semi-Supervised Medical Image Segmentation via Cross Teaching between CNN and Transformer (2021, methodology): “Inspired by (Qiao et al., 2018; Han et al., 2018), we used predictions of unlabeled images generated by CNN/Transformer to update the parameters of the Transformer/CNN respectively.”
Are Anchor Points Really Indispensable in Label-Noise Learning? (2019, methodology): “For example, In this paper, we study the class-dependent and instance-independent transition matrix, i.e., P ( ¯ Y = j | Y = i, X = x ) = P ( ¯ Y = j | Y = i ) , which is identiﬁable under mild conditions and on which the vast majority of current methods focus [14, 13, 30, 29, 26].”
Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression (2021, background): “However, there is little work on the robustness of localization losses to noisy bboxes, even though a number of methods have been proposed for robust learning with noisy labels, anchors, and bboxes [27, 26, 12, 10, 15, 37, 38, 25, 19, 20].”
Learning to Detect and Retrieve Objects From Unlabeled Videos (2019, methodology): “tracted frames lack the object in the scene. Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally overﬁt the data with the noise [7, 8, 29]. Handling extreme noise levels for image classiﬁcation has been shown in [8], (tested on ”
Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis (2019, background): “(b) MentorNet [33], Co-Teaching [22] and NAL [37] are classic weakly-supervised learning methods for training deep networks.”