The State of Sparsity in Deep Neural Networks

By Trevor Gale, Erich Elsen, Sara Hooker. 2019

891 citations recorded by the source corpus.

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Sparsity in the sequence length is an area that allows for training efficiently with extremely long sequences (Zaheer et al., 2020; Tay et al., 2020; Choromanski et al., 2020; Kitaev et al., 2020; Roy et al., 2020; Child et al., 2019; Gale et al., 2019).”
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts (2022, background): “Exploiting sparsity in the weights, activations and input data of deep neural networks (DNNs) is an effective technique for reducing the amount of computation that is needed to achieve a given model quality (Han et al., 2015; Gale et al., 2019).”
A Review of Sparse Expert Models in Deep Learning (2022, background): “These are a special case of sparse neural networks (Gale et al., 2019; Dettmers and Zettlemoyer, 2019; Evci et al., 2020) which are similar in that they only use a subset of parameters, but differ because they have potentially irregular sparsity patterns.”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “Recall that Gale et al. [14] report an apparent dichotomy between sparsity and performance when training (residual) models with unstructured sparsity using the L 0 reparametrization of Louizos et al. [31].”
On the Predictability of Pruning Across Scales (2020, methodology): “, 2015), and produces stateof-the-art tradeoffs between density and error (Gale et al., 2019; Blalock et al., 2020; Renda et al., 2020).”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “This agrees with the observations made in works on model compression: sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization (Zhu and Gupta, 2017; Gale et al., 2019).”
Sparse is Enough in Scaling Transformers (2021, critical): “Model pruning [24, 2] makes matrices smaller by removing unneeded weights after or during training, however, the gains in computational complexity on sparse matrices often do not result in inference speedups on actual hardware [9].”
The Efficiency Misnomer (2021, background): “Unstructured sparse models are models where weights of dense operations are made to contain many (almost) zeros (Gale et al., 2019; Evci et al., 2020), which do not contribute to the operation’s result and thus, in principle, can be skipped.”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “The basic pruning approach removes individual weights from the network (Swayamdipta et al., 2018; Gale et al., 2019).”
What is the State of Neural Network Pruning? (2020, methodology): “This is exceptionally rare, with Gale et al. (2019) and Liu et al. (2019) being arguably the only examples.”
Dynamic Model Pruning with Feedback (2020, methodology): “Following the common experimental setting in related work on network pruning (Liu et al., 2019; Gale et al., 2019; Dettmers & Zettlemoyer, 2019; Mostafa & Wang, 2019), our main experiments focus on ResNet (He et al., 2016a) and WideResNet (Zagoruyko & Komodakis, 2016).”
Continual Learning via Neural Pruning (2019, methodology): “A manifestation of this over-parametrization is through the practice of sparsiﬁca-tion, i.e. the compression of neural network with relatively little loss of performance [4, 11, 18].”
The hardware lottery (2020, background): “For example, unstructured pruning (Hooker et al., 2019; Gale et al., 2019; Evci et al., 2019) and weight speciﬁc quantization (Zhen et al., 2019) are very successful compression techniques in deep neural networks but are incompatible with current hardware and compilation kernels.”
Rigging the Lottery: Making All Tickets Winners (2019, methodology): “Gale et al. (2019) examined magnitude pruning, L 0 Regularization, and Variational Dropout and concluded that they all † Pronounced ”wriggle”. achieve about the same accuracy versus sparsity trade-off on ResNet-50 and Transformer architectures.”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, methodology): “5 For information on other pruning techniques, we recommend Gale et al. (2019) and Liu et al. (2019b).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, background): “For further details, see the survey of Gale et al. (2019).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “However, this approach leads to sparse matrix operations, which are known to be hard to accelerate, and which are typically memory-bound [21, 66].”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “It has been applied to a large variety of architectures in computer vision [Guo et al., 2016], in language processing [Gale et al., 2019], and more recently has been leveraged as a core component in the lottery ticket hypothesis [Frankle et al., 2019].”
Sparse GPU Kernels for Deep Learning (2020, methodology): “Since the topology of sparse matrices in DNNs is typically updated infrequently, the cost of the argsort of the row indices by their row lengths can be amortized over many training steps [6], [13], [17].”