Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation

By Chenze Shao, Yang Feng. 2022

44 citations recorded by the source corpus.

Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. This problem is called catastrophic forgetting, which is a fundamental challenge in the continual learning of neural networks. In this work, we observe that catastrophic forgetting not only occurs in continual learning but also affects the traditional static training. Neural networks, especially neural machine translation models, suffer from catastrophic forgetting even if they learn from a static training set. To be specific, the final model pays imbalanced attention to training samples, where recently exposed samples attract more attention than earlier samples. The underlying cause is that training samples do not get balanced training in each model update, so we name this problem imbalanced training. To alleviate this problem, we propose Complementary Online Knowledge Distillation (COKD), which uses dynamically updated teacher models trained on specific data orders to iteratively provide complementary knowledge to the student model. Experimental results on multiple machine translation tasks show that our method successfully alleviates the problem of imbalanced training and achieves substantial improvements over strong baseline systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Machine Translation, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Hang Su
3. Xingxing Zhang
4. Armen Aghajanyan
5. Liyuan Wang
6. Luke S. Zettlemoyer
7. Ying Shan
8. Aram H. Markosyan

In Computer Vision and Pattern Recognition

1. Ying Shan
2. Hang Su
3. Jun Zhu
4. Xintao Wang
5. Liyuan Wang
6. Xingxing Zhang
7. Armen Aghajanyan
8. Xiaodong Cun

In Radiology, Nuclear Medicine and Imaging

1. Ying Shan
2. Liyuan Wang
3. Jun Zhu
4. Hang Su
5. Xingxing Zhang
6. Xintao Wang

In Machine Translation

1. Luke S. Zettlemoyer

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Simone Scardapane
6. Valerio Marsocci
7. Liyuan Wang
8. Hongwei Yan

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The applications of continual learning in NLP are also highly diverse, such as dialogue system [130], [131], [260], [260], [282], [298], [437], text classification [87], [172], [325], [460], sentence generation [246], [298], [499], relation learning [343], [357], [455], [465], neural machine transla”
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, background): “We mark this score to identify the movement of the generated video.”
• Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models (2022, background): “A number of mechanisms have been proposed for increasing robustness against catastrophic forgetting [28, 29, 30, 31, 32, 33].”