DivideMix: Learning with Noisy Labels as Semi-supervised Learning

By Junnan Li, Richard Socher, Steven C. H. Hoi. 2020

1,389 citations recorded by the source corpus.

Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reducing the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label co-guessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Code is available at this https URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2021, background): “Similar ideas have been explored in semi-supervised learning [35], label noise learning [36], and very recently in contrastive learning [37].”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, methodology): “We then fit a GMM to the training loss of all examples in the memory such that it estimates the probability of an example being pG(·) [22].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Small loss sample selection [36, 72, 45] techniques by co-teaching [85, 21, 26, 90, 55, 11] identify noisy samples with multiple models in the same vein.”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, neutral): “…et al., 2019; Jiang et al., 2015; Liang et al., 2016), cross-validation (Northcutt et al., 2019), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017; Li et al., 2020; Zhang et al., 2020), data augmentation (Zhang et al., 2018; Cheng et al., 2019; 2020; Liang et al., 2020), among others.”
Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, methodology): “DivideMix [101] is a recent technique that trains two networks simultaneously.”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, methodology): “In addition, the comparison with DM-DYR-SH [1] and DivideMix [18], which are in fact not directly comparable with MORPH, is discussed in Appendix B.”
A Survey on Deep Semi-Supervised Learning (2021, methodology): “DivideMix [117] presents a new DSSL framework to improve MixMatch by utilizing co-refinement and co-guessing, and handles the problem of learning with noisy labels.”
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations (2021, methodology): “, 2020] and Divide-Mix [Li et al., 2020] adopt two networks with advanced strategies such as mix-up data augmentation, their performances on CIFAR-100N largely outperforms all other methods.”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “The existing work on other tasks and the popularity in other fields like image classification (Xiao et al., 2015; Li et al., 2017; Lee et al., 2018; Mahajan et al., 2018; Li et al., 2020) suggests, however, that distant supervision could”
MoPro: Webly Supervised Learning with Momentum Prototypes (2020, methodology): “There exists a large body of literature on learning with label noise (Jiang et al., 2018; Han et al., 2018; Guo et al., 2018; Tanaka et al., 2018; Arazo et al., 2019; Li et al., 2020a).”
Early-Learning Regularization Prevents Memorization of Noisy Labels (2020, methodology): “The approach is inspired by Co-teaching and related methods [14, 50, 22].”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, neutral): “…and Mitchell 1998; Yan et al. 2017; Patrini et al. 2016; Zhang and Sabuncu 2018; Xu et al. 2019; Menon et al. 2020; Ma et al. 2020) or inexplicitly used in experiments for synthetizing noisy labels (Han et al. 2018b; Yu et al. 2019; Arazo et al. 2019; Li, Socher, and Hoi 2020; Lukasik et al. 2020).”
Robust Multi-View Clustering With Incomplete Information (2022, background): “optimization; and iv) semi-supervised learningmethods [27], [37], which first identify the clean samples from the noisy ones and then optimize the network by treating the clean samples as labeled and the noisy ones as unlabeled.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…could be: correcting mislabeled examples present in a dataset (Arazo et al., 2019), using them only in a self-supervised training objective (Li et al., 2020a), or even completely ignoring them during training (Wang et al., 2018; Talukdar et al., 2021), upweighting or balancing training on…”
Understanding and Improving Early Stopping for Learning with Noisy Labels (2021, methodology): “We use the ipping rate including correct labels in Table 3 to compare with results in [16], and”
Mutual consistency learning for semi-supervised medical image segmentation (2021, methodology): “…the average surface distance (ASD) and the 95% Hausdorﬀ Distance (95HD) for the quantitative evaluation. by UA-MT (Yu et al., 2019), SASSNet (Li et al., 2020b), DTC (Luo et al., 2021a), URPC (Luo et al., 2021a), MC-Net (Wu et al., 2021), and our MC-Net+ model on the Pancreas-CT dataset,…”
An Overview of Deep Semi-Supervised Learning (2020, background): “Other works [35, 69, 87, 96] have shown that SSL can be useful in learning from noisy labels, where the noisy labels are discarded, and the noisy examples are considered as unlabeled data and used to regularize training using SSL methods.”