Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation

By Huda Khayrallah, Brian Thompson, Kevin Duh, Philipp Koehn. 2018

64 citations recorded by the source corpus.

Supervised domain adaptation—where a large generic corpus and a smaller in-domain corpus are both available for training—is a challenge for neural machine translation (NMT). Standard practice is to train a generic model and use it to initialize a second model, then continue training the second model on in-domain data to produce an in-domain model. We add an auxiliary term to the training objective during continued training that minimizes the cross entropy between the in-domain model’s output word distribution and that of the out-of-domain model to prevent the model’s output from differing too much from the original out-of-domain model. We perform experiments on EMEA (descriptions of medicines) and TED (rehearsed presentations), initialized from a general domain (WMT) model. Our method shows improvements over standard continued training by up to 1.5 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Machine Translation

Citations in the corpus

Continual Training of Language Models for Few-Shot Learning (2022, methodology): “It has been used for slot ﬁlling (Shen et al., 2019), language learning (Li et al., 2019), sentence embedding (Liu et al., 2019a), translation (Khayrallah et al., 2018), cross-lingual modeling (Liu et al., 2020b), question answering (Greco et al., 2019) and text classiﬁcation (Ke et al., 2021a,b;…”
Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning (2021, background): “For example, CL models have been proposed for sentiment analysis [23, 24, 37, 43], dialogue slot filling [53], language modeling [58, 7], language learning [31], sentence embedding [33], machine translation [25], cross-lingual modeling [35], and question answering [12].”
Continual Learning for Neural Machine Translation (2021, methodology): “• Knowledge Distillation (KD) (Khayrallah et al., 2018) When ﬁne-tuning on current set of data, this model optimizes a weighted sum of NLL loss and regularization term: The regularization term is formulated in the spirit of knowledge distillation that minimizes the cross-entropy between in-domain…”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “Khayrallah et al. (2018) pointed out that, due to over-fitting, some amount of knowledge learned from the out-of-domain corpus is being forgotten during fine-tuning.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Thompson et al. (2019), Barone et al. (2017), and Khayrallah et al. (2018) propose regularization-based methods that introduce an additional loss to the original ob-jective to help the model trade off between the general-domain and in-domain.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, methodology): “(2017) and Khayrallah et al. (2018), but for the purpose of retaining generaldomain performance as opposed to improving indomain performance.”
Domain Adaptive Inference for Neural Machine Translation (2019, methodology): “Approaches to NMT domain adaptation include training data selection or generation (Sennrich et al., 2016a; Wang et al., 2017; Sajjad et al., 2017) and fine-tuning output distributions (Dakwale and Monz, 2017; Khayrallah et al., 2018).”
Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, background): “For future work, it would be interesting to measure how curriculum learning models perform on the general domain test set (rather than the indomain test set we focus on in this work); do they suffer more or less from catastrophic forgetting (Goodfellow et al., 2014; Kirkpatrick et al., 2017; Khayral”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, methodology): “This ﬁnding may explain the recent success of techniques which regularize a continued training model using the initial, out-of-domain model (Miceli Barone et al., 2017; Dakwale and Monz, 2017; Khayrallah et al., 2018).”
Neural Machine Translation: A Review (2019, methodology): “[489], Dakwale and Monz [490] regularized the output distributions using techniques inspired by knowledge distillation (Sec.”
Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, methodology): “Regularization-based methods (Dakwale and Monz, 2017; Thompson et al., 2019; Barone et al., 2017; Khayrallah et al., 2018) instead introduce an additional loss to the original objective so that the translation model can trade off between general-domain and in-domain.”
Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, methodology): “• Model Distillation (Khayrallah et al. 2018): We employed an auxiliary loss during fine-tuning to prevent the target domain model’s output from differing too much from the original general domain model’s output.”