Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads

By Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, Maosong Sun. 2020

36 citations recorded by the source corpus.

Deep pre-trained Transformer models have achieved state-of-the-art results over a variety of natural language processing (NLP) tasks. By learning rich language knowledge with millions of parameters, these models are usually overparameterized and significantly increase the computational overhead in applications. It is intuitive to address this issue by model compression. In this work, we propose a method, called Single-Shot Meta-Pruning, to compress deep pre-trained Transformers before fine-tuning. Specifically, we focus on pruning unnecessary attention heads adaptively for different downstream tasks. To measure the informativeness of attention heads, we train our Single-Shot Meta-Pruner (SMP) with a meta-learning paradigm aiming to maintain the distribution of text representations after pruning. Compared with existing compression methods for pre-trained models, our method can reduce the overhead of both fine-tuning and inference. Experimental results show that our pruner can selectively prune 50% of attention heads with little impact on the performance on downstream tasks and even provide better text representations. The source code will be released in the future.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Maosong Sun
4. Xu Han
5. Jie Tang
6. Xipeng Qiu
7. Xiaoyan Zhu
8. Ji-rong Wen

In Computer Vision and Pattern Recognition

1. Zhiyuan Liu
2. Maosong Sun
3. Minlie Huang
4. Yang Liu
5. Xu Han
6. Ji-rong Wen
7. Jian Guan
8. Xiaoyan Zhu

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “In (Michel et al., 2019; Voita et al., 2019), and (Zhang et al., 2021b), researchers study the redundancy of the attention heads in Transformers and find that only a small part of them is enough for good performance.”
• CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, background): “Besides, it is important to reduce the model size by model compression (Sanh et al., 2019; Jiao et al., 2019; Zhang et al., 2020).”