On the Predictability of Pruning Across Scales

By Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, Nir Shavit. 2020

48 citations recorded by the source corpus.

We show that the error of iteratively-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We functionally approximate the error of the pruned networks, showing that it is predictable in terms of an invariant tying width, depth, and pruning level, such that networks of vastly different sparsities are freely interchangeable. We demonstrate the accuracy of this functional approximation over scales spanning orders of magnitude in depth, width, dataset size, and sparsity. We show that the scaling law functional form holds (generalizes) for large scale data (CIFAR-10, ImageNet), architectures (ResNets, VGGs) and iterative pruning algorithms (IMP, SynFlow). As neural networks become ever larger and more expensive to train, our findings suggest a framework for reasoning conceptually and analytically about pruning.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Explaining neural scaling laws (2021, background): “Throughout this work we will be interested in how the average test loss L ( D, P ) depends on the dataset size D and the number of model parameters P .”
Reproducible Scaling Laws for Contrastive Language-Image Learning (2022, background): “5, 21 [64] Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “For example, the works in [159] and [160] propose magnitude-based iterative global pruning methods.”