Structured Pruning of Large Language Models

By Ziheng Wang, Jeremy Wohlwend, Tao Lei. Conference on Empirical Methods in Natural Language Processing, 2019

354 citations recorded by the source corpus.

Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study this question through the lens of model compression. We present a novel, structured pruning approach based on low rank factorization and augmented Lagrangian L0 norm regularization. Our structured approach achieves significant inference speedups while matching or outperforming our unstructured pruning baseline at various sparsity levels. We apply our method to state of the art models on the enwiki8 dataset and obtain a 1.19 perplexity score with just 5M parameters, vastly outperforming a model of the same size trained from scratch. We also demonstrate that our method can be applied to language model fine-tuning by pruning the BERT model on several downstream classification benchmarks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/83b8108014e3db4f46354a28ae68193f143c4e7e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, neutral): “It is usually divided into three types: unstructured ([39, 46, 47]), semi-structured (also called pattern-based) ([54, 55, 56]) and structured ([57, 58, 59]).”
A Comprehensive Overview of Large Language Models (2023, neutral): “While the user can provide in-context samples in the query [86], [85], here we specifically refer to the methods that access external storage programmatically, calling them augmented LLMs.”
MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models (2021, methodology): “For structured pruning, [11, 34] uses LayerDrop to train the model and observes that small/efficient models can be extracted from the pre-trained model; [46] uses a low-rank factorization of the weight matrix and adaptively removes rank-1 components during training; and [27] tests head drop for mult”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “Masks can also be learned from data, with an objective function that balances model performance and network sparsity (Louizos, Welling, and Kingma, 2018; Wang, Wohlwend, and Lei, 2020; Cao, Sanh, and Rush, 2021).”
A Survey on Vision Transformer (2020, methodology): “[131] decompose the standard matrix multiplication in the transformer models and achieve more efficient inference.”
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (2023, background): “…(Figure 1) that there are abundant structures observed in attention modules (Michel et al., 2019; Voita et al., 2019; Clark et al., 2019; Wang et al., 2020; Child et al., 2019), and not all attention modules need to attend to all tokens (Liu et al., 2023b; Zhang et al., 2023; Liu et…”
AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search (2020, methodology): “In Wang et al. (2019b), low-rank factorization is adopted to prune a BERT variation, RoBERTa (Liu et al., 2019c), for several classiﬁcation tasks.”
Block Pruning For Faster Transformers (2021, methodology): “While Mao et al. (2020) combine SVD-based matrix factorization with un-structured pruning, Wang et al. (2019) use structured pruning in order to reduce the rank.”
Effective Sparsification of Neural Networks with Global Sparsity Constraint (2021, positive): “Network Pruning [10, 8, 39, 21, 24, 14, 41, 17, 34, 30, 38] has been extensively studied in recent years to reduce the model size and improve the inference efﬁciency of deep neural networks.”
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (2023, methodology): “…often overparameterized and can be pruned significantly with minimal impact on performance (Han et al., 2016; Wen et al., 2016; Liu et al., 2017; Luo et al., 2017; Cai et al., 2019; Deng et al., 2020; Hou et al., 2020; Wang et al., 2020; Lagunas et al., 2021; Xia et al., 2022; Kurtic et al., 2023).”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, methodology): “There is a line of works dedicated to improving the inference efficiency of PLMs, including pruning (Wang et al., 2020; Gordon et al., 2020), low-rank factorization (Ma et al., 2019; Noach and Goldberg, 2020; Lan et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Kim et al.,…”
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, methodology): “FLOP decomposes weight matrices of a network into the sum of rank-1 components, which are regularized during training to gain sparsity.”
LLM-Pruner: On the Structural Pruning of Large Language Models (2023, methodology): “Several techniques are proposed to solve these problems, like model pruning [53, 56, 64, 20], knowledge distillation [43, 38, 44],quantization [1, 12] within the context of pre-trained language model (PLM).”
Pruning a BERT-based Question Answering Model (2019, methodology): “egularization to BERT while focusing on linguistic interpretability of attention heads but did not report speedups. L 0 regularization was combined with matrix factorization to prune transformers in (Wang et al., 2019). Gale et al. (2019) induced unstructured sparsity on a transformer-based MT model”
Learned Token Pruning for Transformers (2021, methodology): “As mentioned in Section 3.3, the training procedure of LTP consists of two stages: soft pruning that trains both the model parameters and thresholds on downstream tasks, followed by hard pruning that fine-tunes the model parameters with fixed thresholds.”