Improving Generalization by Controlling Label-Noise Information in Neural Network Weights

By Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, Aram Galstyan. 2020

61 citations recorded by the source corpus.

In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. If one considers neural network weights as random variables that depend on the data and stochasticity of training, the amount of memorized information can be quantified with the Shannon mutual information between weights and the vector of all training labels given inputs, $I(w : \mathbf{y} \mid \mathbf{x})$. We show that for any training algorithm, low values of this term correspond to reduction in memorization of label-noise and better generalization bounds. To obtain these low values, we propose training algorithms that employ an auxiliary network that predicts gradients in the final layers of a classifier without accessing labels. We illustrate the effectiveness of our approach on versions of MNIST, CIFAR-10, and CIFAR-100 corrupted with various noise models, and on a large-scale dataset Clothing1M that has noisy labels.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tongliang Liu
2. Dacheng Tao
3. P. Heng
4. Bo Han
5. Chris Dongjoo Kim
6. Fengxiang He
7. Guangyong Chen
8. Pengfei Chen

In Computational Mechanics

1. Tongliang Liu
2. Dacheng Tao
3. Chen Gong
4. Yi Chang
5. P. Heng
6. Gunhee Kim
7. Bo Han
8. Sang-chul Moon

Citations in the corpus

• Continual Learning on Noisy Data Streams via Self-Purified Replay (2021, background): “Several works design the noise corrected losses [86, 28, 46, 4, 84] so that the loss minimization of the whole data becomes similar to that of clean samples.”
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021, unknown): “[13] have previously analyzed phenomena similar to (1), and 230 here we demonstrate that this issue really does occur for the models/datasets widely used in current 231 practice.”
• Noise against noise: stochastic label noise helps combat inherent label noise (2021, methodology): “The finding motivates several robust learning methods (Harutyunyan et al., 2020; Xie et al., 2020) and also supports our method.”
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting (2020, methodology): “With this approach, Theorem 2.1 of Harutyunyan et al. (2020) also supported that memorization of noisy labels is prevented by decreasing I(θ; y|x).”
• Robust early-learning: Hindering the memorization of noisy labels (2021, background): “It would be interesting to explore the potential characteristics of the updated parameters such as the distance to initial parameters (Li et al., 2020b; Hu et al., 2020) and the mutual information with the vector of all training labels given inputs (Harutyunyan et al., 2020).”