Learning from Noisy Labels with Distillation

By Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Li. 2017

593 citations recorded by the source corpus.

The ability of learning from noisy labels is very useful in many visual recognition tasks, as a vast amount of data with noisy labels are relatively easy to obtain. Traditionally, label noise has been treated as statistical outliers, and techniques such as importance re-weighting and bootstrapping have been proposed to alleviate the problem. According to our observation, the real-world noisy labels exhibit multimode characteristics as the true labels, rather than behaving like independent random outliers. In this work, we propose a unified distillation framework to use “side” information, including a small clean dataset and label relations in knowledge graph, to “hedge the risk” of learning from noisy labels. Unlike the traditional approaches evaluated based on simulated label noises, we propose a suite of new benchmark datasets, in Sports, Species and Artifacts domains, to evaluate the task of learning from noisy labels in the practical setting. The empirical study demonstrates the effectiveness of our proposed method in all the domains.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Learning Visual N-Grams from Web Data (2016, methodology): “To circumvent this problem, several recent studies have tried to use image data from photo-sharing websites such as Flickr to train their models [5, 9, 19, 28, 41, 46, 57, 58, 62]: such images have no manually curated annotations, but they do have metadata such as tags, captions, comments, and geolo”
Similarity-Preserving Knowledge Distillation (2019, background): “We believe that similarity-preserving distillation can provide a simple yet effective drop-in replacement for (or complement to) traditional forms of distillation in a variety of application areas, including model compression [28], privileged learning [21], adversarial defense [25], and learning wit”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Techniques that repair labels [39, 80, 50, 74, 27, 56] or directly learn them [76, 88] are also viable options for learning from noisy labeled data.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation (2022, background): “…et al., 2017); memory aware synapses (MAS) (Aljundi et al., 2018) uses the gradients of the model outputs; and episodic memory or gradient episodic memory (GEM) (Li et al., 2017; Lopez-Paz and Ranzato, 2017) allows positive backward transfer and prevents the loss on past tasks from increasing.”
Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Knowledge distillation has soon been applied to a variety of tasks, including image classification (Hinton et al., 2015; Furlanello et al., 2018; Yang et al., 2018; Anil et al., 2018; Li et al., 2017), speech recognition (Hinton et al.”
Label Refinery: Improving ImageNet Classification through Label Progression (2018, background): “[15] uses cross-category relationships from knowledge graphs to mitigate the issues caused by noisy labels.”
Learning to Reweight Examples for Robust Deep Learning (2018, methodology): “…community (Natarajan et al., 2013; Angluin & Laird, 1988) and practical methods have also been proposed (Reed et al., 2014; Sukhbaatar & Fergus, 2014; Xiao et al., 2015; Azadi et al., 2016; Goldberger & Ben-Reuven, 2017; Li et al., 2017; Jiang et al., 2017; Vahdat, 2017; Hendrycks et al., 2018).”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, background): “[22] proposed a unified framework to distill the knowledge from clean labels and knowledge graph, which can be exploited to learn a better model from noisy labels.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Model distillation [3, 94, 148, 173, 189, 200, 260, 262, 280] is a method in which a large model with higher accuracy is used as a teacher to help the training of a compact student model.”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, methodology): “Another method that uses knowledge distillation for noisy labels has also been proposed [22].”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…(a) formulating explicit or implicit noise models to characterize the distribution of noisy and true labels using neural networks (Goldberger and BenReuven 2017; Jiang et al. 2018; Sukhbaatar et al. 2015), graphical models (Xiao et al. 2015; Li et al. 2017), etc. and (b) using correction methods.”
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise (2018, methodology): “This assumption has been leveraged by others for the purpose of label noise robustness, most notably [24, 9, 25, 19].”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, methodology): “To verify MentorNet on real-world noisy labels, we conduct experiments on the large WebVision benchmark (Li et al., 2017a).”
Improving Streaming Automatic Speech Recognition with Non-Streaming Model Distillation on Unsupervised Data (2020, methodology): “(3) We use the waveforms and their predicted transcripts in a noisy student learning framework [14, 15, 16, 17].”
Unifying semi-supervised and robust learning by mixup (2019, methodology): “Some work of RLL uses p > 0 setting explicitly (Charikar et al. (2017); Li et al. (2017); Hendrycks et al. (2018)) and implicitly (Jiang et al. (2018); Jenni & Favaro (2018)).”
Normalized Loss Functions for Deep Learning with Noisy Labels (2020, methodology): “Here, we test the effectiveness of our APL loss functions on large-scale real-world noisy dataset WebVision (Li et al., 2017a).”
Towards Understanding Knowledge Distillation (2019, background): “, 2017), learning from noisy labels (Li et al., 2017), defending against adversarial attacks (Papernot et al.”
How does Disagreement Help Generalization against Label Corruption? (2019, background): “[15] proposed a uniﬁed framework to distill the knowledge from clean labels and knowledge graph, which can be exploited to learn a better model from noisy labels.”
Learning to Learn From Noisy Labeled Data (2018, background): “One line of approaches formulate explicit or implicit noise models to characterize the distribution of noisy and true labels, using neural networks [5, 8, 11, 19, 16, 23, 29], directed graphical models [31], knowledge graphs [13], or conditional random fields [27].”