Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

By Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, Philipp Koehn. 2019

146 citations recorded by the source corpus.

Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading in-domain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, methodology): “EWC has previously been successfully applied to neural machine translation (NMT), where Thompson et al. (2019) and Saunders et al. (2019) minimize catastrophic forgetting when adapting the domain of models through fine-tuning.”
Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022, background): “There are also a number of efforts to address the catastrophic forgetting problem for the domain adaptation of NMT. Kirkpatrick et al. (2017); Thompson et al. (2019) added regularization terms to constrain the update of parameters.”
Continual Learning for Neural Machine Translation (2021, background): “Saunders et al. (2019) and Thompson et al. (2019) add an L2 or EWC regularization to each parameter to prevent the model’s parameters from changing too much.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “In the works of Miceli Barone et al. (2017); Khayrallah et al. (2018); Variš and Bojar (2019); Thompson et al. (2019) regularization approaches (e.g. EWC) were leveraged.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Thompson et al. (2019), Barone et al. (2017), and Khayrallah et al. (2018) propose regularization-based methods that introduce an additional loss to the original ob-jective to help the model trade off between the general-domain and in-domain.”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, methodology): “3462 applied to the domain adaptation task for Neural Machine Translation (Thompson et al., 2019; Saunders et al., 2019).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, methodology): “EWC is applied as described in Thompson et al. (2019). The final submission is an ensemble decode of the four”
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, positive): “One effective form is Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) which in NMT has been shown to maintain or even improve original domain performance (Thompson et al., 2019; Saunders et al., 2019).”
Domain Adaptive Inference for Neural Machine Translation (2019, methodology): “Concurrently with our work, Thompson et al. (2019) apply EWC to reduce forgetting during NMT domain adaptation.”
Continual learning in recurrent neural networks (2020, background): “Indeed, some studies report promising results in the context of natural language processing (NLP) (Madasu and Rao, 2020; Thompson et al., 2019), while others ﬁnd that it performs poorly (Asghar et al., 2020; Cossu et al., 2020a; Li et al., 2020).”
Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, background): “Regularization meth-ods often integrate extra training objectives to prevent parameters from large deviations, such as model output regularization (Khayrallah et al. 2018), elastic weight consolidation (EWC) (Thompson et al. 2019).”
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, methodology): “Compared with Thompson et al. (2019), our work is better at dealing with large domain divergence, since we add domain-specific parts to the model.”
Adversarial target-invariant representation learning for domain generalization (2019, background): “Despite the success of domain adaptation algorithms in several tasks (Chattopadhyay et al., 2012; Serdyuk et al., 2016; Thompson et al., 2019), such techniques rely on the assumption that an unlabeled sample from a target distribution of interest is known at training time.”
Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “forgetting problem in various domains, such as machine translation (Miceli-Barone et al., 2017; Thompson et al., 2019) and reading comprehension (Xu et al.”