Learning to Learn From Noisy Labeled Data

By Junnan Li, Yongkang Wong, Qi Zhao, Mohan Kankanhalli. 2018

372 citations recorded by the source corpus.

Despite the success of deep neural networks (DNNs) in image classification tasks, the human-level performance relies on massive training data with high-quality manual annotations, which are expensive and time-consuming to collect. There exist many inexpensive data sources on the web, but they tend to contain inaccurate labels. Training on noisy labeled datasets causes performance degradation because DNNs can easily overfit to the label noise. To overcome this problem, we propose a noise-tolerant training algorithm, where a meta-learning update is performed prior to conventional gradient update. The proposed meta-learning method simulates actual training by generating synthetic noisy labels, and train the model such that after one gradient update using each set of synthetic noisy labels, the model does not overfit to the specific noise. We conduct extensive experiments on the noisy CIFAR-10 dataset and the Clothing1M dataset. The results demonstrate the advantageous performance of the proposed method compared to several state-of-the-art baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “…the data during training (Reed et al. 2015), jointly optimizing the model’s parameters and estimating true labels (Tanaka et al. 2018), using noise-tolerant loss function (Ghosh, Kumar, and Sastry 2017; Van Rooyen, Menon, and Williamson 2015) or noise tolerant training algorithms (Li et al. 2019).”
MoPro: Webly Supervised Learning with Momentum Prototypes (2020, methodology): “Second, many methods perform computation-heavy procedures for noise cleaning (Jiang et al., 2018; Li et al., 2019; 2020a), or require access to a set of samples with clean labels (Vahdat, 2017; Veit et al.”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, methodology): “amples by erasing their effect on model via scaled gradient ascent. Model-agnostic metalearning (MAML) [146] seeks for weight initialization which can easily be ﬁne-tuned. Similar approach is used in [109] for noisy labels, which aims to ﬁnd noise-tolerant model parameters that are less prone to noi”
Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation (2021, background): “On the other hand, knowledge distillation (KD) [25, 32, 33, 65] is proven effective to transfer clean knowledge from the teacher model to the student even when the network learns from itself [10, 32].”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, background): “[68] Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S.”
Noise against noise: stochastic label noise helps combat inherent label noise (2021, background): “7) One can combat label noise with reﬁned training strategies (Li et al., 2019b; 2020; Nguyen et al., 2020) that potentially incorporate several techniques, including sample selection/weighting, label correction, meta-learning (Li et al., 2019b) and semi-supervised learning (Tarvainen & Valpola,…”
Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization (2020, methodology): “Note that we do not use the 50k clean training data in all the experiments because only noisy labels are required during the training process [20, 35].”
Deep learning with noisy labels: exploring techniques and remedies in medical image analysis (2019, background): “Along the same lines, [155] proposed a meta-learning objective that encouraged consistent predictions between a student model trained on noisy labels and a teacher model trained on clean labels.”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…noise), in which the label of each example is independently and uniformly changed to a random (incorrect) class with a probability (Zhang et al., 2017; Arpit et al., 2017; Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018; Li et al., 2019; Arazo et al., 2019).”
Deep Self-Learning From Noisy Labels (2019, background): “[16] tried to solve noisy label training by meta-learning.”
Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark (2022, neutral): “Accurate object detection in the label noise scenario is of great interest both in the computer vision and earth vision community (Li et al., 2020a, 2019a; Pelletier et al., 2017).”
Learning From Noisy Anchors for One-Stage Object Detection (2019, background): “It has been demonstrated that the outputs from networks can indicate the noise level of samples when labels are corrupted and noisy for image classification tasks—the network tends to learn clean samples quickly early on and make confident predictions for them, while recognizing noisy samples slowly”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, background): “…label is also commonly used as a benchmark to evaluate robust learning methods in a noise-control setting e.g. in (Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018b; Van Rooyen et al., 2015; Li et al., 2019; Arazo et al., 2019; Charoenphakdee et al., 2019).”
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights (2020, background): “Li et al. (2019) propose a meta-learning approach that optimizes a classification loss along with a consistency loss between predictions of a mean teacher and predictions of the model after a single gradient descent step on a synthetically labeled mini-batch.”