Online Distilling from Checkpoints for Neural Machine Translation

By Hao-Ran Wei, Shujian Huang, Ran Wang, Xinyu Dai, Jiajun Chen. 2019

34 citations recorded by the source corpus.

Current predominant neural machine translation (NMT) models often have a deep structure with large amounts of parameters, making these models hard to train and easily suffering from over-fitting. A common practice is to utilize a validation set to evaluate the training process and select the best checkpoint. Average and ensemble techniques on checkpoints can lead to further performance improvement. However, as these methods do not affect the training process, the system performance is restricted to the checkpoints generated in original training procedure. In contrast, we propose an online knowledge distillation method. Our method on-the-fly generates a teacher model from checkpoints, guiding the training process to obtain better performance. Experiments on several datasets and language pairs show steady improvement over a strong self-attention-based baseline system. We also provide analysis on data-limited setting against over-fitting. Furthermore, our method leads to an improvement in a machine reading experiment as well.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Xiaodan Liang
3. Jianping Gou
4. Liang Lin
5. S. Maybank
6. Baosheng Yu
7. Frederick Liu
8. Chenze Shao

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Xiaodan Liang
3. Liang Lin
4. Luc Van Gool
5. Xiaogang Wang
6. Ming-Hsuan Yang
7. Trevor Darrell
8. Yi Yang

Citations in the corpus

• Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, methodology): “We compare our method with knowledge distillation methods and deep mutual learning (Zhang et al., 2018), and also report the results of Online Distillation from Checkpoints (ODC) (Wei et al., 2019) for comparison.”
• Knowledge Distillation: A Survey (2020, methodology): “To obtain lightweight NMT, there are many extended knowledge distillation methods for neural machine translation (Hahn and Choi 2019; Zhou et al. 2019a; Kim and Rush 2016; Gordon and Duh 2019; Wei et al. 2019; Freitag et al. 2017; Tan et al. 2019).”
• Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks (2021, background): “Another famous regularization-based approach is knowledge distillation [64] that preserves the prediction produced by the”
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation (2020, background): “Wei et al. (2019) introduced a novel training procedure where there is no need for an external teacher.”
• Towards Quantifiable Dialogue Coherence Evaluation (2021, background): “In recent years, KD has been applied to many specific tasks (Sun et al., 2020; Wei et al., 2019; Kim and Rush, 2016; Sourty et al., 2020).”
• Estimating Training Data Influence by Tracking Gradient Descent (2020, methodology): “Checkpoint ensembling is a widely used technique in machine translation [12], semi-supervised learning [13] and knowledge distillation [14] which provide intuition on why TracIn performs better than other methods.”