Extreme Compression for Pre-trained Transformers Made Simple and Efficient

By Xiaoxia Wu, Zi-Mei Yao, Minjia Zhang, Conglong Li, Yuxiong He. 2022

31 citations recorded by the source corpus.

Extreme compression, particularly ultra-low bit precision (binary/ternary) quantization, has been proposed to fit large NLP models on resource-constraint devices. However, to preserve the accuracy for such aggressive compression schemes, cutting-edge methods usually introduce complicated compression pipelines, e.g., multi-stage expensive knowledge distillation with extensive hyperparameter tuning. Also, they oftentimes focus less on smaller transformer models that have already been heavily compressed via knowledge distillation and lack a systematic study to show the effectiveness of their methods. In this paper, we perform a very comprehensive systematic study to measure the impact of many key hyperparameters and training strategies from previous works. As a result, we find out that previous baselines for ultra-low bit precision quantization are significantly under-trained. Based on our study, we propose a simple yet effective compression pipeline for extreme compression, named XTC. XTC demonstrates that (1) we can skip the pre-training knowledge distillation to obtain a 5-layer BERT while achieving better performance than previous state-of-the-art methods, e.g., the 6-layer TinyBERT; (2) extreme quantization plus layer reduction is able to reduce the model size by 50x, resulting in new state-of-the-art results on GLUE tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Geophysics, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Kurt Keutzer
2. A. Gholami
3. Michael W. Mahoney
4. Dan Alistarh
5. Elias Frantar
6. Torsten Hoefler
7. Sehoon Kim
8. Saleh Ashkboos

In Computer Vision and Pattern Recognition

1. Kurt Keutzer
2. Michael W. Mahoney
3. A. Gholami
4. Dan Alistarh
5. Torsten Hoefler
6. Elias Frantar
7. Sehoon Kim
8. Jitendra Malik

In Model Architectures

1. A. Gholami
2. Karttikeya Mangalam
3. Jitendra Malik
4. Michael W. Mahoney
5. Elias Frantar
6. Torsten Hoefler
7. Kurt Keutzer
8. Sehoon Kim

In Efficiency & Compression

1. Elias Frantar
2. Dan Alistarh
3. A. Gholami
4. Kurt Keutzer
5. Michael W. Mahoney
6. Torsten Hoefler
7. Saleh Ashkboos
8. Sehoon Kim

Citations in the corpus

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, background): “, 2022), while prior training-based techniques have only tackled models that are smaller by one to two orders of magnitude (Wu et al., 2022).”
• Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”