Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

By Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste. 2021

986 citations recorded by the source corpus.

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “In this work, we follow the network pruning approach, where the redundant units (weights, neurons or ﬁlters/channels) of a pre-trained NN are removed; see [Kuzmin et al., 2019, Blalock et al., 2020, Hoeﬂer et al., 2021] for recent surveys.”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, background): “(Hoefler et al., 2021; Gholami et al., 2021), surprisingly little is known about compressing such models for inference.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “We refer the interested reader to [66, 95, 132] for a thorough survey of related work in pruning/sparsity. d) Knowledge distillation : Model distillation [3, 94, 148, 173, 189, 200, 261, 262] involves training a large model and then using it as a teacher to train a more compact model.”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, neutral): “…instead of the commonly employed 32-bit floating point format; ( 2 ) pruning or sparsity (Molchanov et al., 2016; Liu et al., 2018; He et al., 2019; Hoefler et al., 2021), which aims to eliminate unnecessary neurons or weights within the models; ( 3 ) and distillation (Hinton et al., 2015; Cho &…”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “Both sparsification (Hoefler et al., 2021; Frantar & Alistarh, 2023) and quantization (Kwon et al.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “It is natural to ask whether the truly massive memory costs of such MoEs can be reduced via standard techniques for model compression , such as quantization (Gholami et al., 2021) or sparsity (Hoefler et al., 2021), without significant accuracy loss.”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, background): “A majority of model compression techniques fall into one of four categories: distillation, tensor decomposition (which includes low-rank factorization), pruning and quantization (Hoefler et al., 2021; Gholami et al., 2021; Zhu et al., 2023; Gupta & Agrawal, 2021).”
How Well Do Sparse ImageNet Models Transfer? (2021, background): “Importantly, its compressibility has also emerged as a consistent benchmark for CNN pruning methods [28].”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, positive): “As these outliers are often unstructured, we encode them individually in a row-wise arrangement similar to a compressed-sparse-row (CSR) representation [HABN + 21].”
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts (2024, neutral): “Deep learning models are often dense and over-parameterized (Hoefler et al., 2021), leading to increased memory and computational demands during both training and inference.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “Despite its simplicity, magnitude pruning has been used to find extremely sparse networks [32, 77] and now stands out as a strong baseline approach [3, 23, 28] for neural network sparsification.”
TIES-Merging: Resolving Interference When Merging Models (2023, background): “…in the merged model and eliminate subtle distinctions among values: (1) I NTERFERENCE FROM REDUNDANT PARAMETERS : Previous studies on model pruning [25, 76] have shown that during fine-tuning, many model parameters can change over the course of fine-tuning [63] but only have a small impact on…”
Neuromorphic computing at scale (2025, neutral): “Sparsification can be either structural (that is, weights, neurons, heads) or ephemeral (that is, activations, gradients, errors) 22 .”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “…(Han et al., 2015; Jacob et al., 2018; Nagel et al., 2019; Zhao et al., 2019), pruning (Molchanov et al., 2016; Liu et al., 2018; He et al., 2019; Hoefler et al., 2021), and distillation (Hinton et al., 2015; Cho & Hariharan, 2019; Tang et al., 2019; Touvron et al., 2021) have been applied to…”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “Hundreds of such pruning and quantization approaches have been proposed and analyzed [17, 11], with the general goal of obtaining efﬁcient variants of deep neural nets (DNNs) which would preserve accuracy while maximizing compression.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “The work in [31] provides a comprehensive review of sparsity in deep learning up to 2020, but with little studying on emerging pruning methods, such as pruning in contrastive learning [32] and self-supervised pruning [33], etc.”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, methodology): “This approach has a long history [28, 19], and has been applied successfully in the case of vision and smaller-scale language models and tasks [21].”