LLM-Pruner: On the Structural Pruning of Large Language Models

By Xinyin Ma, Gongfan Fang, Xinchao Wang. Neural Information Processing Systems, 2023

892 citations recorded by the source corpus.

Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in both the deployment, inference, and training stages. With LLM being a general-purpose task solver, we explore its compression in a task-agnostic manner, which aims to preserve the multi-task solving and language generation ability of the original LLM. One challenge to achieving this is the enormous size of the training corpus of LLM, which makes both data transfer and model post-training over-burdensome. Thus, we tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures based on gradient information, maximally preserving the majority of the LLM's functionality. To this end, the performance of pruned models can be efficiently recovered through tuning techniques, LoRA, in merely 3 hours, requiring only 50K data. We validate the LLM-Pruner on three LLMs, including LLaMA, Vicuna, and ChatGLM, and demonstrate that the compressed models still exhibit satisfactory capabilities in zero-shot classification and generation. The code is available at: https://github.com/horseee/LLM-Pruner

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/017010b941d902a467f6d329ae5e74fd67e67912

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, methodology): “For example LLM-pruner (Ma et al., 2023a) removes connected structures from an LLM before further training.”
A Comprehensive Overview of Large Language Models (2023, neutral): “For instance, the 11B model can become competitive to 540B PaLM in [246] and 7.5B to 280B Gopher in [237].”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “It is usually divided into three types: un-structured ([48], [49], [50]), semi-structured (also called pattern-based) ([17], [51], [52]) and structured ([20], [39], [53]).”
A Simple and Effective Pruning Approach for Large Language Models (2023, background): “Recently, Ma et al. (2023) have studied structured pruning of LLMs. Bansal et al. (2023); Liu et al. (2023b) and Elena Voita (2023) have demonstrated the existence of prompt-dependent and task-specific sparsity in the structural components of LLMs, e.g., attention heads and MLP neurons.”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “The pruning ratios ranged from 0% to 97%, employing the strategy of deleting layers based on the importance proposed in this paper, resulting in models with 1 to 32 layers.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “When comparing performance between LLM-Pruner and ours in continued pre-training, our model achieves lower perplexity than LLM-Pruner with a similar”