A Simple and Effective Pruning Approach for Large Language Models

By Mingjie Sun, Zhuang Liu, Anna Bair, J. Kolter. International Conference on Learning Representations, 2023

914 citations recorded by the source corpus.

As their size increases, Large Languages Models (LLMs) are natural candidates for network pruning methods: approaches that drop a subset of network weights while striving to preserve performance. Existing methods, however, require either retraining, which is rarely affordable for billion-scale LLMs, or solving a weight reconstruction problem reliant on second-order information, which may also be computationally expensive. In this paper, we introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs. Motivated by the recent observation of emergent large magnitude features in LLMs, our approach prunes weights with the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. Notably, Wanda requires no retraining or weight update, and the pruned LLM can be used as is. We conduct a thorough evaluation of our method Wanda on LLaMA and LLaMA-2 across various language benchmarks. Wanda significantly outperforms the established baseline of magnitude pruning and performs competitively against recent method involving intensive weight update. Code is available at https://github.com/locuslab/wanda.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/7d22ad3573101337bca2091fb0114b377c4f3db6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, neutral): “For simplicity, K S i , ∗ denotes a sub-matrix of K which selects S i rows from K .”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “Both unstructured pruning, semi-structured pruning (Frantar & Alistarh, 2023; Sun et al., 2023), and structured pruning (Ma et al.”
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (2024, background): “Pruning removes weights from models while preserving model quality and inference performance (Chee et al., 2022; Sun et al., 2023).”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “It is usually divided into three types: un-structured ([48], [49], [50]), semi-structured (also called pattern-based) ([17], [51], [52]) and structured ([20], [39], [53]).”
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (2024, neutral): “Among these techniques, SparseGPT generally delivers the best results, while magnitude-based and WANDA methods produce comparable performance both during pruning and after retraining as shown in Table 6.”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “SparseGPT (Frantar & Alistarh, 2023) applied the same idea for pruning and sparsifies the LLMs using unstructured and semi-structured pruning, and Sun et al. (2023) simplified the idea by using only the diagonal of the Hessian.”