SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

By Elias Frantar, Dan Alistarh. 2023

1,399 citations recorded by the source corpus.

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.

Subfields: Artificial Intelligence, Computer Networks and Communications, Electrical and Electronic Engineering, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Electrical and Electronic Engineering

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “However, for general-purpose LLMs, pruning inevitably re-sults in performance degradation compared to original models (Frantar & Alistarh, 2023; Sun et al., 2023; Ma et al., 2023), especially when without significant compute invested post-pruning.”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, neutral): “…is an optimal point for round-to-nearest-based methods, whereas higher compression can be achieved via data-aware methods such as GPTQ. SparseGPT [FA23] presented an approach to jointly sparsify LLM weights to medium sparsities, together with quantization of the remaining weights to a fixed…”
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, positive): “Among these techniques, SparseGPT generally delivers the best results, while magnitude-based and WANDA methods produce comparable performance both during pruning and after retraining as shown in Table 6.”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “This is a useful comparison to ACDC as learnable masks do not change the weights of our model after pruning (Frantar and Alistarh, 2023).”
Structured information extraction from scientific text with large language models (2024, background): “Methods for reducing the number of parameters needed for LLM inference and ﬁ ne-tuning 56 – 59 are also a promising avenue for reducing the complexity and cost of self-hosting LLMs.”
LLM-Pruner: On the Structural Pruning of Large Language Models (2023, background): “The first term is typically neglected in prior work [23, 51, 11], as the model has already converged on the training”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “Recently, there is active research attempting to apply one or a combination of such techniques in LLM inference (Yao et al., 2022; Park et al., 2022; Dettmers et al., 2022; Frantar et al., 2022; Frantar & Alistarh, 2023; Bansal et al., 2022; Xiao et al., 2022).”
High-throughput Generative Inference of Large Language Models with a Single GPU (2023, methodology): “Both spar-sification (Hoefler et al., 2021; Frantar & Alistarh, 2023) and quantization (Kwon et al., 2022; Yao et al., 2022; Park et al., 2022; Xiao et al., 2022; Frantar et al., 2022; Dettmers et al., 2022) have been adopted for LLM inference.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “Unstructured pruning methods ([13], [17]) remove weights anywhere and can achieve high prune ratios with little impact on accuracy.”
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models (2023, background): “…costs would be affordable enough to execute on such models, are currently only able to reduce precision to 3 or 4 bits per parameter (Frantar et al., 2022; Dettmers & Zettlemoyer, 2022; Wu et al., 2023) or around 50% sparsity (Frantar & Alistarh, 2023), before significant accuracy loss occurs.”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “• SparseGPT [20] is a second-order pruning method based on solving a layer-wise reconstruction problem.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “…future work explores compression techniques (Treviso et al., 2023) such as quantization (Dettmers et al., 2022; Frantar et al., 2022; Ahmadian et al., 2023) and pruning (Frantar and Alistarh, 2023; Ogueji et al., 2022; Gale et al., 2019; Ahia et al., 2021) for multilingual instruction-tuned models.”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “The unstructured pruning simplifies an LLM by removing specific parameters without considering its internal structure, such as SparseGPT Frantar and Alistarh (2023) and LoRAPrune Zhang et al. (2023).”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, positive): “…efforts have employed model compression techniques with specific designs to achieve efficient LLM inference, such as the method described in (Frantar & Alistarh, 2023), which employs one-shot pruning on LLMs (e.g., OPT-175 B and BLOOM-176 B), resulting in negligible performance degradation…”