Learning Light-Weight Translation Models from Deep Transformer

By Bei Li, Ziyang Wang, Hui Liu, Quan Du, Tong Xiao, Chunliang Zhang, Jingbo Zhu. 2020

44 citations recorded by the source corpus.

Recently, deep models have shown tremendous improvements in neural machine translation (NMT). However, systems of this kind are computationally expensive and memory intensive. In this paper, we take a natural step towards learning strong but light-weight NMT systems. We proposed a novel group-permutation based knowledge distillation approach to compressing the deep Transformer model into a shallow model. The experimental results on several benchmarks validate the effectiveness of our method. Our compressed model is 8 times shallower than the deep model, with almost no loss in BLEU. To further enhance the teacher model, we present a Skipping Sub-Layer method to randomly omit sub-layers to introduce perturbation into training, which achieves a BLEU score of 30.63 on English-German newstest2014. The code is publicly available at https://github.com/libeineu/GPKD.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Knowledge Distillation: A Survey (2020, methodology): “…including the design of knowledge (Hinton et al. 2015; Romero et al. 2015) and the loss functions for matching features or distributions matching (Huang and Wang 2017; Passalis and Tefas 2018; Zagoruyko and Komodakis 2017; Mirzadeh et al. 2020; Li et al. 2020d; Heo et al. 2019b; Asif et al. 2020).”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, methodology): “State of art pruning techniques can achieve a far higher level of compression and performance than simply using a smaller dense network (Zhu and Gupta, 2017; Li et al., 2020b).”
Intriguing Properties of Compression on Multilingual Models (2022, background): “While most compression techniques have minimal impact on aggregate performance numbers (Gale et al., 2019; Li et al., 2020; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021), the impact on individual sub-populations in the data, such as low-resource languages, can be…”