Learning to Reweight Examples for Robust Deep Learning

By Mengye Ren, Wenyuan Zeng, Binh Yang, Raquel Urtasun. 2018

1,690 citations recorded by the source corpus.

Deep neural networks have been shown to be very powerful modeling tools for many supervised learning tasks involving complex input patterns. However, they can also easily overfit to training set biases and label noises. In addition to various regularizers, example reweighting algorithms are popular solutions to these problems, but they require careful tuning of additional hyperparameters, such as example mining schedules and regularization hyperparameters. In contrast to past reweighting methods, which typically consist of functions of the cost value of each example, in this work we propose a novel meta-learning algorithm that learns to assign weights to training examples based on their gradient directions. To determine the example weights, our method performs a meta gradient descent step on the current mini-batch example weights (which are initialized from zero) to minimize the loss on a clean unbiased validation set. Our proposed method can be easily implemented on any type of deep network, does not require any additional hyperparameter tuning, and achieves impressive performance on class imbalance and corrupted label problems where only a small amount of clean validation data is available.

Subfields: Artificial Intelligence, Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

In Safety, Robustness & Interpretability

Citations in the corpus

On Robustness of Neural Architecture Search Under Label Noise (2020, methodology): “…methods that aim to correct mislabeled data (Brodley and Friedl, 1999) or adjust the sampling weights of unreliable training instances ( Chang et al., 2017; Han et al., 2018; Jiang et al., 2018; Ren et al., 2018; Yu et al., 2019) (adding Co-teaching from Han et al., 2018 and Yu et al., 2019).”
Exploring Example Influence in Continual Learning (2022, neutral): “Some studies propose a similar idea and use the inﬂuences to reweight or dropout the training data [31, 13, 36].”
Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, methodology): “Notably, SPR works much better than L2R [68], which additionally uses 1000 clean samples for training, giving it a substantial advantage over all the other baselines.”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, methodology): “Closest to our work, bilevel optimization was used in [47] to reweigh samples to overcome training set imbalance or corruption, and sensor selection was approached via bilevel optimization in [51].”
Meta Pseudo Labels (2020, background): “Similar bi-level optimization problems have been proposed to optimize a model’s learning process, such as learning the learning rate schedule [3], designing architectures [40], correcting wrong training labels [88], generating training examples [59], and re-weighting training data [73, 74, 54, 53].”
Decoupling Representation and Classifier for Long-Tailed Recognition (2019, background): “A more ﬁne-grained control of the loss can also be achieved at sample level, e.g . with Focal loss (Lin et al., 2017), Meta-Weight-Net (Shu et al., 2019), re-weighted training (Ren et al., 2018), or based on Bayesian uncertainty (Khan et al., 2019).”
Meta-Learning in Neural Networks: A Survey (2020, background): “samples [147], [148], discount outliers [67], or correct for class imbalance [147].”
Class-Balanced Loss Based on Effective Number of Samples (2019, background): “A side effect of assigning higher weights to hard examples is the focus on harmful samples ( e.g ., noisy data or mislabeled data) [23, 32].”
Co-teaching: Robust training of deep neural networks with extremely noisy labels (2018, background): “[34] leveraged an additional validation set to adaptively assign weights to training examples in every iteration.”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “, 2013), and meta-learning (Ren et al., 2018), to differentiate instances within datasets.”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, background): “[33] used a small clean dataset and put more weights on noisy samples which have gradients closer to that of the clean dataset.”
Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learning (2020, methodology): “isy Data Learning with noisy data can be divided into two approaches: (a) formulating explicit or implicit noise models to characterize the distribution of noisy and true labels using neural networks [14,21,33,60,56,73], graphical models [96,35], etc. and (b) using correction methods. E.g., relabeli”
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise (2018, background): “This assumption has been leveraged by others for the purpose of label noise robustness, most notably [24, 9, 25, 19].”
Unsupervised label noise modeling and loss correction (2019, methodology): “Still other approaches use curriculum learning to formulate a robust learning procedure (Jiang et al., 2018b; Ren et al., 2018).”
Tilted Empirical Risk Minimization (2020, methodology): “, 2018), LearnRewight (Ren et al., 2018), FocalLoss (Lin et al.”
Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (2020, background): “example difficulty [21], estimated Bayesian uncertainty [19], gradient direction [29].”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, neutral): “…(Reed et al., 2014; Goldberger & Ben-Reuven, 2017; Li et al., 2017b; Veit et al., 2017; Song et al., 2019), example weighting (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016), cross-validation (Northcutt et al., 2019), semi-supervised learning…”
DivideMix: Learning with Noisy Labels as Semi-supervised Learning (2020, background): “Another approach selects or reweights samples so that noisy samples contribute less to the loss (Jiang et al., 2018; Ren et al., 2018).”
Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition From a Domain Adaptation Perspective (2020, methodology): “It is worth noting some seemingly small and yet fundamental differences between our algorithm and the learning to re-weight (L2RW) method [43].”