Dimensionality-Driven Learning with Noisy Labels

By Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah Monazam Erfani, Shutao Xia, Sudanthi N. R. Wijewickrema, James Bailey. 2018

475 citations recorded by the source corpus.

Datasets with significant proportions of noisy (incorrect) class labels present challenges for training accurate Deep Neural Networks (DNNs). We propose a new perspective for understanding DNN generalization for such datasets, by investigating the dimensionality of the deep representation subspace of training samples. We show that from a dimensionality perspective, DNNs exhibit quite distinctive learning styles when trained with clean labels versus when trained with a proportion of noisy labels. Based on this finding, we develop a new dimensionality-driven learning strategy, which monitors the dimensionality of subspaces during training and adapts the loss function accordingly. We empirically demonstrate that our approach is highly tolerant to significant proportions of noisy labels, and can effectively learn low-dimensional local subspaces that capture the data distribution.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, background): “Also note that most prior works (Goldberger & Ben-Reuven, 2017; Ma et al., 2018; Han et al., 2018) only experiment with small-scale images since label corruption demonstrations can require training hundreds of models (Hendrycks et al.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Learning with noisy labeled data has long been studied [92, 5, 54, 35].”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, background): “In addition, there are some other deep learning solutions to deal with noisy labels [24, 41].”
Unsupervised label noise modeling and loss correction (2019, neutral): “the criterion followed in previous experiments), a criterion adopted by several other authors (Jiang et al., 2018b; Ma et al., 2018; Ren et al., 2018; Wang et al., 2018b).”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…noise), in which the label of each example is independently and uniformly changed to a random (incorrect) class with a probability (Zhang et al., 2017; Arpit et al., 2017; Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018; Li et al., 2019; Arazo et al., 2019).”
Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise (2020, unknown): “Using predictions of DNNs has long been adopted in distillation (Hinton, Vinyals, and Dean 2015) and robust training algorithms that use pseudo labels (Reed et al. 2015; Ma et al. 2018; Tanaka et al. 2018; Song, Kim, and Lee 2019; Nguyen et al. 2019; Arazo et al. 2019).”
Robust Inference via Generative Classifiers for Handling Noisy Labels (2019, methodology): “(g) D2L (Ma et al., 2018): Training with new labels generated by a convex combination of the noisy labels and their predictions, where its weights are chosen by utilizing the Local Intrinsic Dimensionality (LID).”
Towards Fair and Privacy-Preserving Federated Deep Models (2019, background): “However, large-scale datasets are not always available for a new domain, due to the signiﬁcant time and effort required for data collection and annotation [2], [3].”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, methodology): “F-correction [26] reweights the forward or backward loss of the training samples based on the label transition matrix estimated using a pre-trained DNN. D2L [23] employs a simple measure called local intrinsic dimensionality and then uses it to modify the forward loss in order to reduce the effects ”
Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis (2019, methodology): “Others methods try to adjust loss functions [37]–[39].”
Robust early-learning: Hindering the memorization of noisy labels (2021, background): “These settings are widely used in existing works (Ma et al., 2018; Thulasidasan et al., 2019; Pleiss et al., 2020; Wang et al., 2021a).”
Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels (2019, methodology): “Another approach proposes to correct labels using the predictions of DNNs, e.g., Bootstrap (Reed et al., 2015), Joint Optimization (Tanaka et al., 2018) and D2L (Ma et al., 2018), all of which are vulnerable to overﬁtting.”
Identifying Mislabeled Data using the Area Under the Margin Ranking (2020, background): “Boot-strap (Reed et al., 2014) and D2L (Ma et al., 2018) interpolate the one-hot target label with the predicted label.”
NLNL: Negative Learning for Noisy Labels (2019, methodology): “Additional label-cleaning methods exist, that gradually change the data label to the prediction value of the network [26, 30, 22].”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “Other types of synthetic label noise were also proposed, including class-conditional noises (Patrini et al., 2017; Rolnick et al., 2017; Reeve & Kab´an, 2019; Han et al., 2018a), noises from other datasets (Wang et al., 2018; Seo et al., 2019), etc .”
Part-dependent Label Noise: Towards Instance-dependent Label Noise (2020, background): “Speciﬁcally, RCN assumes that clean labels ﬂip randomly with a constant rate [1, 4, 24]; CCN assumes that the ﬂip rate depends on the latent clean class [40, 18, 75]; IDN considers the most general case of label noise, where the ﬂip rate depends on its instance.”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, background): “[63] showed that when training on data with noisy labels, the local dimensionality of the features initially decreases as the model learns the dominant patterns in the data.”