A Survey on Model Compression for Large Language Models

By Xunyu Zhu, Jian Li, Yong Liu, Can Ma, Weiping Wang. Transactions of the Association for Computational Linguistics, 2023

484 citations recorded by the source corpus.

Abstract Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/338d8f3b199abcebc85f34016b0162ab3a9d5310

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding (2024, unknown): “Acceleration solutions do exist to deploy to commodity GPUs on lap-tops but they suffer from significant drop in accuracy Zhu et al. (2023).”
A Comprehensive Overview of Large Language Models (2023, unknown): “1: The trends in the number of LLM models introduced over the years.”
SliceGPT: Compress Large Language Models by Deleting Rows and Columns (2024, background): “A majority of model compression techniques fall into one of four categories: distillation, tensor decomposition (which includes low-rank factorization), pruning and quantization (Hoefler et al., 2021; Gholami et al., 2021; Zhu et al., 2023; Gupta & Agrawal, 2021).”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “The training of LLMs is a computationally intensive procedure that requires substantial hardware and energy resources [178].”
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “To alleviate the demanding hardware requirements for deploying massive trained models, many researchers have focused on model compression techniques Zhu et al. (2023) that reduce the inference cost.”
A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, unknown): “In addition, cognitive psychology has been studied for a long time, so many effective human memory theories and architectures have been accumulated, which can support more advanced capabilities of the agents [147].”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “ACDC is different from pruning and other compression techniques (Zhu et al., 2023) since i) the compressed networks we find are reflective of the circuits that model’s use to compute outputs to certain tasks (Section 4) and ii) our goal is not to speed up forward passes, and generally our…”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Some recent review works (such as [35], [40]) survey compression techniques for large models, encompassing a broad array of topics from pruning, quantization, knowledge distillation, etc.”