MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

By Lu Jiang, Zhengyuan Zhou, Thomas K. Leung, Li-Jia Li, Li Fei-Fei. 2017

1,683 citations recorded by the source corpus.

Recent deep networks are capable of memorizing the entire data even when the labels are completely random. To overcome the overfitting on corrupted labels, we propose a novel technique of learning another neural network, called MentorNet, to supervise the training of the base deep networks, namely, StudentNet. During training, MentorNet provides a curriculum (sample weighting scheme) for StudentNet to focus on the sample the label of which is probably correct. Unlike the existing curriculum that is usually predefined by human experts, MentorNet learns a data-driven curriculum dynamically with StudentNet. Experimental results demonstrate that our approach can significantly improve the generalization performance of deep networks trained on corrupted training data. Notably, to the best of our knowledge, we achieve the best-published result on WebVision, a large benchmark containing 2.2 million images of real-world noisy labels. The code are at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…methods that aim to correct mislabeled data (Brodley and Friedl, 1999) or adjust the sampling weights of unreliable training instances ( Chang et al., 2017; Han et al., 2018; Jiang et al., 2018; Ren et al., 2018; Yu et al., 2019) (adding Co-teaching from Han et al., 2018 and Yu et al., 2019).”
Online Continual Learning on a Contaminated Data Stream with Blurry Task Boundaries (2022, background): “MentorNet [16] maintains a teacher network to provide the student network with the smallloss examples.”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “∗Equal Contribution training procedures [85, 36].”
An Empirical Study of Example Forgetting during Deep Neural Network Learning (2018, methodology): “Fan et al. (2017); Kim & Choi (2018); Jiang et al. (2018) learn a curriculum directly from data in order to minimize the task loss.”
Meta-Learning in Neural Networks: A Survey (2020, background): “process and select examples of the right difficulty by defining a teaching policy as the meta-knowledge and training it to optimize the student’s progress [146], [151].”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, methodology): “Among those works, the representative methods are MentorNet [17] and Decoupling [26].”
Unsupervised label noise modeling and loss correction (2019, background): “Loss correction approaches (Reed et al., 2015; Jiang et al., 2018b; Patrini et al., 2017; Zhang et al., 2018) modify either the loss directly, or the probabilities used to compute it, to compensate for the incorrect guidance provided by the noisy samples.”
From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation (2021, background): “This small-loss trick , which is based on the memorization effect [3, 42] of deep networks, has also adopted by other works [17, 30] to select more reliable labels.”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “Recent methods have shown successes in overcoming training data bias such as label noise [3, 16, 23, 24, 52, 40, 49, 66] and biased data distributions [56, 10, 35, 38, 9].”
Are Anchor Points Really Indispensable in Label-Noise Learning? (2019, methodology): “For example, many state-of-the-art approaches in this category are speciﬁcally designed to, e.g., select reliable examples [45, 14, 24], reweight examples [33, 15], correct labels [23, 17, 37, 32], employ side information [39, 21], and (implicitly) add regularization [13, 12, 43, 39, 21].”
MetaCorrection: Domain-aware Meta Loss Correction for Unsupervised Domain Adaptation in Semantic Segmentation (2021, positive): “Authors in [19, 37] utilized multiple layer perception network to automatically assign large weighting factor for easy samples.”
MoPro: Webly Supervised Learning with Momentum Prototypes (2020, methodology): “The main reasons include: (1) most methods do not consider OOD samples, which commonly occur in real-world web datasets; (2) most methods are computational-heavy due to co-training (Han et al., 2018; Li et al., 2020a; Jiang et al., 2018; 2020), iterative training (Tanaka et al.”
Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers (2019, background): “The state-of-the-art is based on selecting the clean data instances in the train set in order to train the network satisfactorily [9, 10, 11].”
Denoising Implicit Feedback for Recommendation (2020, background): “Similar conclusions about noisy samples are also found in the computer vision domain [7, 12, 38], further demonstrating that the phenomenon that noisy samples are harder to fit in the early training stages isn’t model-specific, data-specific, or even domain-specific.”
Dimensionality-Driven Learning with Noisy Labels (2018, background): “An alternative is to correct noisy labels to their true labels via a clean label inference step (Vahdat, 2017; Veit et al., 2017; Jiang et al., 2017; Li et al., 2017).”