Beyond neural scaling laws: beating power law scaling via data pruning

By Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos. Neural Information Processing Systems, 2022

656 citations recorded by the source corpus.

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/45122c8f76a4e2fd0163d1f0522db37e97ea4721

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scaling Data-Constrained Language Models (2023, background): “Alternative data strategies Large pre-training datasets are commonly ﬁltered to remove undesired samples or reduce noise [101].”
The MiniPile Challenge for Data-Efficient Language Models (2023, methodology): “In preliminary BERT training runs, we also tried selecting only the top-l documents closest to the centroid of their assigned clusters, which one may interpret as excluding hard-to-learn outlier examples [54].”
SemDeDup: Data-efficient learning at web-scale through semantic deduplication (2023, methodology): “It is possible that results may change with scale, though following [13], it is likely increasing scale would further improve the beneﬁts of data curation.”
Large Language Models (2023, background): “This point of view suggests that the problem will eventually be solved with larger models and perhaps better training protocols which focus on the more informative data items [126].”
Data Selection for Language Models via Importance Resampling (2023, neutral): “Following RoBERTa [48],”
DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023, methodology): “Interestingly, for iterated DoReMi at the 280M scale, the second round weights achieve slightly worse downstream accuracy than the round 1 weights when used to train 280M models, but transfer better to training 8B models.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…corpus of data (Zhang, 1992; Bien and Tibshirani, 2012; Kim et al., 2015; Kim et al., 2016), which can be used to speed up training (Sener and Savarese, 2018; Shim et al., 2021; Huggins et al., 2017; Sorscher et al., 2022) or aid in interpretability of the model predictions (Yoon et al., 2019).”
DataComp: In search of the next generation of multimodal datasets (2023, background): “[126] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S.”