An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation

By Chenhui Chu, Raj Dabre, Sadao Kurohashi. 2017

240 citations recorded by the source corpus.

In this paper, we propose a novel domain adaptation method named “mixed fine tuning” for neural machine translation (NMT). We combine two existing approaches namely fine tuning and multi domain NMT. We first train an NMT model on an out-of-domain parallel corpus, and then fine tune it on a parallel corpus which is a mix of the in-domain and out-of-domain corpora. All corpora are augmented with artificial tags to indicate specific domains. We empirically compare our proposed method against fine tuning and multi domain methods and discuss its benefits and shortcomings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, methodology): “Our work gets inspiration from a practice in Johnson et al. (2016) and Chu et al. (2017) based on appending domain tags.”
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Chu et al. (2017) ﬁne tunes the model with the mix of the general-domain data and over-sampled in-domain data.”
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, methodology): “Chu et al. (2017) found that mixing general-domain data with the in-domain data used for continued training improved generaldomain performance of the resulting models, at the expense of training time.”
Regularized Training Objective for Continued Training for Domain Adaptation in Neural Machine Translation (2018, neutral): “Chu et al. (2017) present mixed ﬁne-tuning.”
Dynamic Data Selection for Neural Machine Translation (2017, methodology): “Other work combines fine-tuning with model ensembles (Freitag and AlOnaizan, 2016) or with domain-specific tags in the training corpus (Chu et al., 2017). Finally, Sennrich et al. (2016a) adapt their systems by backtranslating in-domain data, which is then added to the training data and used for fin”
Extreme Adaptation for Personalized Neural Machine Translation (2018, methodology): “Domain adaptation techniques for MT often rely on data selection (Moore and Lewis, 2010; Li et al., 2010; Chen et al., 2017; Wang et al., 2017), tuning (Luong and Manning, 2015; Miceli Barone et al., 2017), or adding domain tags to NMT input (Chu et al., 2017).”
Improving Domain Adaptation Translation with Domain Invariant and Specific Information (2019, methodology): “Chu et al. (2017) construct the training data set for the NMT model by combining out-of-domain data with the over-sampled in-domain data.”
Token-level Adaptive Training for Neural Machine Translation (2020, unknown): “• Sampler (Chu et al., 2017).”
Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, background): “These work can be grouped into two categories: data-centric and model ﬁne-tuning (Chu and Wang 2018).”
Learning Hidden Unit Contribution for Adapting Neural Machine Translation Models (2018, methodology): “Chu et al. (2017) propose a reﬁnement of this method where the in-domain data is mixed with the out-of-domain data.”
A Survey of Domain Adaptation for Neural Machine Translation (2018, methodology): “(2018) conduct a study that applys mixed fine tuning (Chu et al., 2017) on synthetic parallel data (Sennrich et al.”
Improving Gender Translation Accuracy with Filtered Self-Training (2021, unknown): “In order to train these models, we ﬁne-tune the base-line using the feminine and masculine samples, and additionally mix in an equal number of sentences from the original training corpus to avoid catastrophic forgetting (following the mixed ﬁne-tuning approach of Chu et al., 2017).”
Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination (2018, methodology): “Contrastingly, Farajian et al. (2017) utilized the similarity between each test sentence and the training instances to dynamically set the hyper-parameters of the learning algorithm and update the generic model on the fly.”
Generalized Data Augmentation for Low-Resource Translation (2019, methodology): “After data augmentation, we follow the pre-train and ﬁne tune paradigm for learning (Zoph et al., 2016; Nguyen and Chiang, 2017 mixed ﬁne-tuning strategy of Chu et al. (2017), ﬁne-tuning the base model on the concatenation of the base and augmented datasets.”
Bi-Directional Neural Machine Translation with Synthetic Parallel Data (2018, methodology): “Following the practice of mixed ﬁne-tuning proposed by Chu et al. (2017), we continue training baseline models on augmented data as shown in B-5* of Table 2.”
Neural Machine Translation (2020, unknown): “, 2014) and pre-ordering models (de Gispert et al., 2015), and so on. For instance, the joint translation and language model by Devlin et al. (2014) was influential since it showed large quality improvements on top of a very competitive statistical machine translation system.”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, background): “Based on this factor, out-domain adaptation has been widely studied for MT, ranging from data selection (Li et al., 2010; Wang et al., 2017), tuning (Luong and Man-ning, 2015; Farajian et al., 2017) to domain tags (Chu et al., 2017).”