What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation

By Vitaly Feldman, Chiyuan Zhang. 2020

650 citations recorded by the source corpus.

Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman (2019) proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given. In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in (Feldman, 2019).

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, methodology): “Various formulations of memorization in deep neural networks have been studied in previous papers (Carlini et al., 2019, 2020; Feldman and Zhang, 2020; Zhang et al., 2021).”
Deduplicating Training Data Makes Language Models Better (2021, methodology): “In computer vision, memorization of training data has been studied from various angles for both discriminative and generative models (e.g. Arpit et al., 2017; Webster et al., 2019; Feldman and Zhang, 2020; Stephenson et al., 2021; Teterwak et al., 2021).”
Extracting Training Data from Large Language Models (2020, background): “The tradeoffs between privacy and utility of models have been studied extensively: differentially-private training typically prevents models from capturing the long tails of the data distribution and thus hurts utility [19,20,67].”
Foundation Models and Fair Use (2023, neutral): “…in instance attribution tend to suffer from difficulties in scaling due to high computational cost (e.g., leave-k-out retraining can be costly) (Feldman & Zhang, 2020; Zhang et al., 2021) or being inaccurate or erroneous when applied to complex but realistic model classes (Basu et al., 2020;…”
Modifying Memories in Transformer Models (2020, background): “In general, without speciﬁcally focusing on language models, Feldman (2020); Feldman and Zhang (2020) have demonstrated both theoretical results and empirical evidences to imply that close-to-optimal generalization requires memorization of labels for samples from the low-frequency sub-populations.”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, methodology): “In subsequent work, they develop an approach for approximating memorization using inﬂuence functions Feldman & Zhang (2020).”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, methodology): “We achieve top-1 accuracies of 93.68% on CIFAR-10, 72.80% on CIFAR-100, and 73.94% on ImageNet.”
Datamodels: Predicting Predictions from Training Data (2022, methodology): “Feldman & Zhang (2020) find some empirical support for this hypothesis by identifying memorized images on CIFAR-100 and ImageNet and showing that removing them hurts overall generalization.”
LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, methodology): “Our formulation of influence has been used in identifying mislabeled examples (Pruthi et al., 2020), analyzing memorization effects (Feldman & Zhang, 2020), and deriving various inter-pretability insights (Madsen et al., 2022), but it does not permit counterfactual reasoning.”
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, positive): “However, recent work has established a more symbiotic relationship between memorization and generalization in deep learning [2, 3, 4].”
TRAK: Attributing Model Behavior at Scale (2023, positive): “Further, as Ilyas et al. (2022) observes, datamodels can be viewed as a regularized version of empirical influences (Feldman & Zhang, 2020), which are defined as a difference-in-means estimator, τ(zj)i = ES′∼D[f(zj ; θ⋆(S′))|zi ∈ S′]− ES′∼D[f(zj ; θ⋆(S′))|zi ̸∈ S′] (24) where D is the uniform…”
SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, methodology): “Methods such as SSL Prototypes [13] and memorization [15] search for semantically non-redundant data subsets to train on.”
Beyond neural scaling laws: beating power law scaling via data pruning (2022, methodology): “Pruning scores were EL2N [10] for CIFAR-10 and SVHN and memorization [13] for ImageNet.”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “This fact, combined with the ability of large models to memorize their training data [9,10,22], raises questions about the originality of diffusion outputs.”
Membership Inference Attacks From First Principles (2021, background): “Feldman and Zhang [13] show that not all examples are equal: some examples (“outliers”) have an outsized effect on the learned model when inserted into a training dataset, compared to other (“inlier”) examples.”
Membership Inference Attacks against Generative Models with Probabilistic Fluctuation (positive): “In contrast, deep learning models inevitably exhibit phenomena of memorization [Feldman and Zhang, 2020].”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, background): “Contributions of Long-Tailed Training Examples Instead of identifying mislabeled samples, easy/difficult-to-learn samples from the training set, more theoretical works on detecting the long-tail examples and outliers [28, 52, 53].”