Compression of Generative Pre-trained Language Models via Quantization

By Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, Ngai Wong. 2022

119 citations recorded by the source corpus.

The increasing size of generative Pre-trained Language Models (PLMs) have greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the homogeneous word embeddings caused by reduced capacity and the varied distribution of weights. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rate on GPT-2 and BART, respectively.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Xiao Liu
3. Yuxiao Dong
4. Ming Ding
5. Minjia Zhang
6. Dacheng Tao
7. Zhengxiao Du
8. Yuxiong He

In Large Language Models

1. Jie Tang
2. Ming Ding
3. Xiao Liu
4. Dacheng Tao
5. Mehdi Rezagholizadeh
6. Yuxiao Dong
7. Aohan Zeng
8. Yufei Xue

In Efficiency & Compression

1. Minjia Zhang
2. Jie Tang
3. Yuxiong He
4. Conglong Li
5. Xiaoxia Wu
6. Reza Yazdani Aminabadi
7. Zi-Mei Yao
8. Xiao Liu

Citations in the corpus

• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “, 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel et al.”
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, methodology): “Recently, [61] also uses knowledge distillation to compress GPT-2 models on task-specific problems to INT2.”
• A Survey on Knowledge Distillation of Large Language Models (2024, background): “The performance of the GPT-J-based Toolformer surpasses OPT (66B) (Zhang et al., 2022) and GPT3 (175B) (Brown et al., 2020) greatly.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “1: The trends in the number of LLM models introduced over the years.”
• A Survey of Large Language Models (2023, unknown): “With some loss in model performance, quantized language models have smaller model sizes and can achieve faster inference speed [81, 191, 192].”
• DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, methodology): “, 2019), quantization (Tao et al., 2022; Prato et al., 2020), and knowledge distillation (Hinton et al.”