Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation

By Shuhao Gu, Yang Feng. 2020

29 citations recorded by the source corpus.

Neural machine translation (NMT) models usually suffer from catastrophic forgetting during continual training where the models tend to gradually forget previously learned knowledge and swing to fit the newly added data which may have a different distribution, e.g. a different domain. Although many methods have been proposed to solve this problem, we cannot get to know what causes this phenomenon yet. Under the background of domain adaptation, we investigate the cause of catastrophic forgetting from the perspectives of modules and parameters (neurons). The investigation on the modules of the NMT model shows that some modules have tight relation with the general-domain knowledge while some other modules are more essential in the domain adaptation. And the investigation on the parameters shows that some parameters are important for both the general-domain and in-domain translation and the great change of them during continual training brings about the performance decline in general-domain. We conducted experiments across different language pairs and domains to ensure the validity and reliability of our findings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Machine Translation, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Shuhao Gu
6. Yang Feng
7. Wanying Xie
8. Tianyu Pang

In Computer Vision and Pattern Recognition

1. Hang Su
2. Jun Zhu
3. Liyuan Wang
4. Xingxing Zhang
5. Shuhao Gu
6. Yang Feng
7. Wanying Xie
8. Yinpeng Dong

In Machine Translation

1. Yang Feng
2. Shuhao Gu

In Meta & Continual Learning

1. Xingxing Zhang
2. Liyuan Wang
3. Hang Su
4. Jun Zhu
5. Simone Scardapane
6. Valerio Marsocci
7. Liyuan Wang
8. Hongwei Yan

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The applications of continual learning in NLP are also highly diverse, such as dialogue system [130], [131], [260], [260], [282], [298], [437], text classification [87], [172], [325], [460], sentence generation [246], [298], [499], relation learning [343], [357], [455], [465], neural machine transla”
• Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, background): “C L ] 1 4 A pr 2 02 1 et al., 2019; Voita et al., 2019; Gu and Feng, 2020), we find that only some important parameters in a well-trained NMT model play an important role when generating the translation and unimportant parameters can be erased without affecting the translation quality too much.”