Understanding Knowledge Distillation in Non-autoregressive Machine Translation

By Chunting Zhou, Graham Neubig, Jiatao Gu. 2019

237 citations recorded by the source corpus.

Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually rely on the technique of knowledge distillation, which creates the training data from a pretrained autoregressive model for better performance. Knowledge distillation is empirically useful, leading to large gains in accuracy for NAT models, but the reason for this success has, as of yet, been unclear. In this paper, we first design systematic experiments to investigate why knowledge distillation is crucial to NAT training. We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data. Furthermore, a strong correlation is observed between the capacity of an NAT model and the optimal complexity of the distilled data for the best translation quality. Based on these findings, we further propose several approaches that can alter the complexity of data sets to improve the performance of NAT models. We achieve the state-of-the-art performance for the NAT-based models, and close the gap with the autoregressive baseline on WMT14 En-De benchmark.

Subfields: Artificial Intelligence, Information Systems, Efficiency & Compression, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Graham Neubig
  4. Noah A. Smith
  5. Percy Liang
  6. Quoc V. Le
  7. Jianfeng Gao
  8. Yoshua Bengio

In Information Systems

  1. Ji-rong Wen
  2. Wayne Xin Zhao
  3. Dawn Song
  4. Dacheng Tao
  5. Marine Carpuat
  6. X. Kong
  7. Weijia Xu
  8. Kevin Gimpel

In Efficiency & Compression

  1. Kurt Keutzer
  2. A. Gholami
  3. Michael W. Mahoney
  4. Sehoon Kim
  5. Dacheng Tao
  6. Chitwan Saharia
  7. Jiatao Gu
  8. William Chan

In Machine Translation

  1. Dacheng Tao
  2. Marine Carpuat
  3. Weijia Xu
  4. X. Kong
  5. Ankur Bapna
  6. Orhan Firat
  7. Jiatao Gu
  8. Yung-Sung Chuang

Citations in the corpus