Fast Domain Adaptation for Neural Machine Translation

By Markus Freitag, Yaser Al-Onaizan. 2016

207 citations recorded by the source corpus.

Neural Machine Translation (NMT) is a new approach for automatic translation of text from one human language into another. The basic concept in NMT is to train a large Neural Network that maximizes the translation performance on a given parallel corpus. NMT is gaining popularity in the research community because it outperformed traditional SMT approaches in several translation tasks at WMT and other evaluation tasks/benchmarks at least for some language pairs. However, many of the enhancements in SMT over the years have not been incorporated into the NMT framework. In this paper, we focus on one such enhancement namely domain adaptation. We propose an approach for adapting a NMT system to a new domain. The main idea behind domain adaptation is that the availability of large out-of-domain training data and a small in-domain training data. We report significant gains with our proposed method in both automatic metrics and a human subjective evaluation metric on two language pairs. With our adaptation method, we show large improvement on the new domain while the performance of our general domain only degrades slightly. In addition, our approach is fast enough to adapt an already trained system to a new domain within few hours without the need to retrain the NMT model on the combined data which usually takes several days/weeks depending on the volume of the data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “aging out-of-domain datasets (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016), and (ii) Multilingual NMT, to improve the translation quality on low resource languages via co-training with similar languages (Dong et al.”
Complete Multilingual Neural Machine Translation (2020, positive): “Adding a New Language We further investigate how a cMNMT model behaves when ﬁne-tuned (Freitag and Al-Onaizan, 2016) to a new language.”
Six Challenges for Neural Machine Translation (2017, background): “A currently popular approach is to train a general domain system, followed by training on in-domain data for a few epochs (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016).”
Continual Learning for Neural Machine Translation (2021, background): “In the ﬁeld of natural language processing, there are some researches on solving catastrophic forgetting problem in lifelong learning (Freitag and Al-Onaizan, 2016; Khayrallah et al., 2018; Saun-ders et al., 2019; Thompson et al., 2019).”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Likewise, other works (e.g. Freitag and Al-Onaizan, 2016; Chu et al., 2017) supported this claim.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Gu et al. (2020) finds that the NMT tends to generate more high-frequency tokens and less low-frequency tokens than reference.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, background): “One downside of continued training is that the adapted model’s ability to translate generaldomain sentences is severely degraded during adaptation (Freitag and Al-Onaizan, 2016).”
Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation (2018, background): “15 ) Freitag and Al-Onaizan (2016) found that that ensembling an out-of-domain model with a model trained via continued training can signiﬁcantly reduce the performance drop on the original domain compared to the continued training model alone.”
Dynamic Data Selection for Neural Machine Translation (2017, methodology): “…technique, see Figure 1b, is inspired by the success of domain-speciﬁc ﬁne-tuning (Luong and Manning, 2015; Zoph et al., 2016; Sennrich et al., 2016a; Freitag and Al-Onaizan, 2016), in which a model trained on a large general-domain bitext is trained for a few additional epochs only on small…”
An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation (2017, methodology): “The conventional domain adaptation method is ﬁne tuning , in which an out-of-domain model is further trained on in-domain data (Luong and Manning, 2015; Sen-nrich et al., 2016b; Servan et al., 2016; Freitag and Al-Onaizan, 2016).”
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, positive): “Instead we propose treating gender debiasing as a domain adaptation problem, since NMT models can very quickly adapt to a new domain (Freitag and Al-Onaizan, 2016).”
Domain Adaptive Inference for Neural Machine Translation (2019, methodology): “Training from scratch on each new domain is impractical, while continuing training on a new domain can cause catastrophic forgetting of previous tasks (French, 1999), even in an ensemble (Freitag and Al-Onaizan, 2016).”
Translationese as a Language in “Multilingual” NMT (2019, background): “Overall, tagging can be seen as domain adaptation (Freitag and Al-Onaizan, 2016; Luong and Manning, 2015).”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, methodology): “Continued training (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016), also referred to as fine-tuning, is one of the most popular methods for NMT adaptation, due to its strong performance.”
Synthetic Data for Neural Machine Translation of Spoken-Dialects (2017, methodology): “We tried two approaches to utilize the synthetic data: adding it to the training data as usual and adapting one of the two ensembles by continuing to train it on the synthetic data for 2 more epochs, similar to the approach proposed in [23].”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, methodology): “Continued training or fine-tuning (Luong et al., 2015; Freitag and Al-Onaizan, 2016; Chu et al., 2017) is an adaptation technique where a model is first trained on the large general domain data, then used as initialization of a new model which is further trained on in-domain bitext.”