Compression of Neural Machine Translation Models via Pruning

By Abigail See, Minh-Thang Luong, Christopher D. Manning. 2016

224 citations recorded by the source corpus.

Neural Machine Translation (NMT), like many other deep learning domains, typically suffers from over-parameterization, resulting in large storage sizes. This paper examines three simple magnitude-based pruning schemes to compress NMT models, namely class-blind, class-uniform, and class-distribution, which differ in terms of how pruning thresholds are computed for the different classes of weights in the NMT architecture. We demonstrate the efficacy of weight pruning as a compression technique for a state-of-the-art NMT system. We show that an NMT model with over 200 million parameters can be pruned by 40% with very little performance loss as measured on the WMT'14 English-German translation task. This sheds light on the distribution of redundancy in the NMT architecture. Our main result is that with retraining, we can recover and even surpass the original performance with an 80%-pruned model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “One popular approach to the analysis of NMT representations is to evaluate how informative they are for various linguistic tasks.”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Structured pruning has been used in some NLP applications, such as machine translation (See et al., 2016), text classification (Joulin et al.”
SYSTRAN's Pure Neural Machine Translation Systems (2016, methodology): “This approach has been proven efﬁcient for the NMT tasks in See et al. (2016).”
The Description Length of Deep Learning models (2018, methodology): “…a student layer to approximate a well-trained network (Ba and Caruana, 2014; Romero et al., 2015), or pipelines involving retraining, pruning, and quantization of the model weights (Han et al., 2015a,b; Simonyan and Zisserman, 2014; Louizos et al., 2017; See et al., 2016; Ullrich et al., 2017).”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, background): “There are a couple of works attempting to prune RNNs ( e.g ., GRU in Narang et al. (2017) and LSTM in See et al. (2016)).”
Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden (2018, background): “One representative example is neural network pruning,where a fraction of weights are removed from the trained model and the “lottery ticket” is found when the remained weight can still be quickly trained with competitive loss and accuracy [1], [19], [20], [21].”
What Do Compressed Deep Neural Networks Forget (2019, methodology): “…model (model distillation) (Hinton et al., 2015) and ﬁnally pruning by setting a subset of weights or ﬁlters to zero (Louizos et al., 2017; Wen et al., 2016; Cun et al., 1990; Hassibi et al., 1993b; Ström, 1997; Hassibi et al., 1993a; Zhu & Gupta, 2017; See et al., 2016; Narang et al., 2017).”
Fully Quantized Transformer for Machine Translation (2019, background): “More recently, See et al. (2016) show that pruning a fully trained model and then retraining it can increase performance over the original non-pruned model.”
Importance-based Neuron Allocation for Multilingual Neural Machine Translation (2021, methodology): “For example, See et al. (2016) examines three magnitude-based pruning schemes, Zhu and Gupta (2018) demonstrates that large-sparse models outperform comparably-sized small-dense models, and Wang et al. (2020a) improves the utilization efﬁciency of parameters by introducing a rejuvenation approach.”
Balancing Cost and Benefit with Tied-Multi Transformers (2020, background): “Pruning of pre-trained models (See et al., 2016) makes it possible to discard around 80% of the smallest weights of a model without deterioration in inference quality, given it is re-trained with appropriate hyper-parameters after pruning.”
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, methodology): “Weight Pruning We adopt the magnitude-based weight pruning scheme (See et al., 2016), where”
Structured Pruning of Large Language Models (2019, background): “A common approach to model compression is known as weight pruning (Zhu and Gupta, 2017; Han et al., 2015a; See et al., 2016).”
Distilling Neural Networks for Greener and Faster Dependency Parsing (2020, methodology): “Here we explore teacher-student distillation as a means of increasing the efficiency of neural network systems used to undertake a core task in NLP, dependency parsing.”
A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations (2023, unknown): “Magnitude-based criteria can be applied to either unstructured ([48], [49], [153]) or structured pruning ([63], [94], [132]).”
Neural Machine Translation: A Review (2019, background): “• NMT model shrinking and reduced precision (Wu et al., 2016; See et al., 2016; Kim & Rush, 2016; Freitag et al., 2017; Zhang et al., 2018)”