Pruning a BERT-based Question Answering Model

By J. Scott McCarley. 2019

54 citations recorded by the source corpus.

We investigate compressing a BERT-based question answering system by pruning parameters from the underlying BERT model. We start from models trained for SQuAD 2.0 and introduce gates that allow selected parts of transformers to be individually eliminated. Specifically, we investigate (1) reducing the number of attention heads in each transformer, (2) reducing the intermediate width of the feed-forward sublayer of each transformer, and (3) reducing the embedding dimension. We compare several approaches for determining the values of these gates. We find that a combination of pruning attention heads and the feed-forward layer almost doubles the decoding speed, with only a 1.5 f-point loss in accuracy.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yonatan Belinkov
2. Zhiyuan Liu
3. Maosong Sun
4. Hassan Sajjad
5. Fahim Dalvi
6. Nadir Durrani
7. Qun Liu
8. Xin Jiang

Citations in the corpus

• TinyBERT: Distilling BERT for Natural Language Understanding (2019, neutral): “, 2020), pruning (Cui et al., 2019; McCarley, 2019; F. et al., 2020; Elbayad et al., 2020; Gordon et al., 2020; Hou et al., 2020) or quantization (Shen et al.”
• When BERT Plays the Lottery, All Tickets Are Winning (2020, neutral): “This explains the success of BERT compression studies (Sanh et al., 2019; Jiao et al., 2019; McCarley, 2019; Lan et al., 2020).”
• TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Instead of directly being used for compression, knowledge distillation can also be used in combination with other compression methods like pruning (McCarley, 2019; Mao et al., 2020), low-rank approximation (Mao et al.”
• DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “There have been some attempts to compress and accelerate inference of the Transformer-based models using low-rank approximation [15, 12], weight-sharing [4, 12], knowledge distillation [21, 23, 10, 28], quantization [1, 33, 22, 8] and pruning [16, 17, 3, 25].”
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “Besides, many studies show that for further model compression, KD is usually used to work with other compression techniques such as pruning McCarley (2019), low-rank approximation Mao et al. (2020) and dynamic networks training Hou et al. (2020).”
• Know What You Don't Need: Single-Shot Meta-Pruning for Attention Heads (2020, methodology): “Besides L0 Norm, we adopt the attention head pruning method introduced by Michel et al. (2019), called Head Importance Score for Pruning (HISP).”
• Adversarial Self-Supervised Data Free Distillation for Text Classification (2020, background): “Recent researches (McCarley, 2019; Gordon et al., 2020; Michel et al., 2019) focus on compressing the large-scale models to a shallow and resource-efficient network via weight pruning (Guo et al.”
• Poor Man's BERT: Smaller and Faster Transformer Models (2020, background): “Some notable approaches include strategies to prune parts of the network after training (Michel et al., 2019a; Voita et al., 2019b; McCarley, 2019), reduction through weight factorization and sharing (Lan et al., 2019), compression through knowledge-distillation (Sanh et al., 2019) and quantization…”