Early-Learning Regularization Prevents Memorization of Noisy Labels

By Sheng Liu, Jonathan Niles-Weed, Narges Razavian, C. Fernandez‐Granda. Neural Information Processing Systems, 2020

772 citations recorded by the source corpus.

We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase, before eventually memorizing the examples with false labels. We prove that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, even in simple linear models, and give a theoretical explanation in this setting. Motivated by these findings, we develop a new technique for noisy classification tasks, which exploits the progress of the early learning phase. In contrast with existing approaches, which use the model output during early learning to detect the examples with clean labels, and either ignore or attempt to correct the false labels, we take a different route and instead capitalize on early learning via regularization. There are two key elements to our approach. First, we leverage semi-supervised learning techniques to produce target probabilities based on the model outputs. Second, we design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels. The resulting framework is shown to provide robustness to noisy annotations on several standard benchmarks and real-world datasets, where it achieves results comparable to the state of the art.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/c9b08639a28ab70a06ba9a09eaae98b2fe3dc6c9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

LEGO: Latent Execution-Guided Reasoning for Multi-Hop Question Answering on Knowledge Graphs (2021, background): “Inspired by prior ﬁndings that deep neural networks can ﬁt the correct labels more easily than the noisy ones (Arpit et al., 2017; Zhang et al., 2017; Liu et al., 2020), we propose stochastic hard EM (Min et al., 2019; Chen et al., 2020b; Liang et al., 2018), i.e. , for each question, we only…”
Deep Learning Through the Lens of Example Difficulty (2021, background): “While early-exit strategies in computer vision (Teerapittayanon et al., 2016; Huang et al., 2018) and NLP (Dehghani et al., 2018; Liu et al., 2020b; Schwartz et al., 2020; Xin et al., 2020) suggest predictions for easier examples require less computation, the above example difficulty notions do not…”
Understanding and Improving Early Stopping for Learning with Noisy Labels (2021, methodology): “Recently, based on this phenomenon, many methods [9, 26, 27, 16, 17] have been proposed and achieved promising performance.”
On Memorization in Probabilistic Deep Generative Models (2021, background): “We argue, however, that memorization in supervised learning differs fundamentally from that in generative models, as the label prediction task affects the training dynamics, and label noise is known to induce memorization in supervised learning [23, 24].”
Robust early-learning: Hindering the memorization of noisy labels (2021, background): “Fortunately, even though deep networks will ﬁt all the labels eventually, they ﬁrst ﬁt data with clean labels, which helps generalization (Arpit et al., 2017; Han et al., 2018b; Yu et al., 2019; Liu et al., 2020).”
Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations (2021, methodology): “Note that both ELR+ [Liu et al., 2020] and Divide-Mix [Li et al.”
Regularizing Generative Adversarial Networks under Limited Data (2021, methodology): “Speciﬁcally, we compute our regularization with constant anchors by setting 1) α R =1 and α F = − 1 following the LS-GAN [43] 2) α F = − 0 .”
Robust Learning by Self-Transition for Handling Noisy Labels (2020, result): “Furthermore, compared with prior studies on the memorization of DNNs [2, 19, 22], this is the first work to solve the problem of finding the best transition point.”