Deep Learning on a Data Diet: Finding Important Examples Early in Training

By M. Paul, Surya Ganguli, Gintare Karolina Dziugaite. Neural Information Processing Systems, 2021

744 citations recorded by the source corpus.

Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores -- the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores -- and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples -- we investigate how the data distribution shapes the loss surface and identify subspaces of the model's data representation that are relatively stable over training.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a6e25ca9ee9d3e45c6d1957c0dc3324a9816c34e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Beyond neural scaling laws: beating power law scaling via data pruning (2022, methodology): “Indeed some recent works [9, 10, 11] have demonstrated this possibility by suggesting various metrics to sort training examples in order of their difficulty or importance, ranging from easy or redundant examples to hard or important ones, and pruning datasets by retaining some fraction of the hardes”
DataComp: In search of the next generation of multimodal datasets (2023, background): “More recent efforts in subset selection often operate on already curated datasets [91, 132, 123, 12, 29, 99]”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, methodology): “These methods have been shown to quickly degrade in performance in the presence of even small amounts of noise since upweighting noisy samples hurts generalization (Hu et al., 2021; Paul et al., 2021).”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, positive): “Supervised finetuning of LLMs has increasingly been torn between improving traditional discriminative benchmarks like HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020) and training LLMs to follow instructions, acquire conversational abilities, and be harmless (Askell et al., 2021a).”
SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, background): “Recent approaches to achieve data efﬁciency in deep learning have operated in a supervised setting by deﬁning and ﬁnding “hard” examples not easily learned by partially or fully trained (ensembles of) models [27, 28, 29, 15, 30].”