Neural Machine Translation Training in a Multi-Domain Scenario

By Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Yonatan Belinkov, Stephan Vogel. 2017

47 citations recorded by the source corpus.

In this paper, we explore alternative ways to train a neural machine translation system in a multi-domain scenario. We investigate data concatenation (with fine tuning), model stacking (multi-level fine tuning), data selection and multi-model ensemble. Our findings show that the best translation quality can be achieved by building an initial system on a concatenation of available out-of-domain data and then fine-tuning it on in-domain data. Model stacking works best when training begins with the furthest out-of-domain data and the model is incrementally fine-tuned with the next furthest domain and so on. Data selection did not give the best results, but can be considered as a decent compromise between training time and translation quality. A weighted ensemble of different individual models performed better than data selection. It is beneficial in a scenario when there is no time for fine-tuning an already trained model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Tagged Back-Translation (2019, background): “…to data selection, as in van der Wees et al. (2017); Axelrod et al. (2011), or curriculum learning approaches like ﬁne-tuning on in-domain data (Thompson et al., 2018; Hassan Sajjad and Nadir Durrani and Fahim Dalvi and Yonatan Belinkov and Stephan Vogel, 2017; Freitag and Al-Onaizan, 2016).”
Domain Adaptive Inference for Neural Machine Translation (2019, methodology): “Approaches to NMT domain adaptation include training data selection or generation (Sennrich et al., 2016a; Wang et al., 2017; Sajjad et al., 2017) and fine-tuning output distributions (Dakwale and Monz, 2017; Khayrallah et al., 2018).”
A Survey of Domain Adaptation for Neural Machine Translation (2018, methodology): “…and Zong, 2016b; Cheng et al., 2016; Currey et al., 2017; Domhan and Hieber, 2017), synthetic corpora (Sennrich et al., 2016b; Zhang and Zong, 2016b; Park et al., 2017), or parallel copora (Chu et al., 2017; Sajjad et al., 2017; Britz et al., 2017; Wang et al., 2017a; van der Wees et al., 2017).”
Reinforcement Learning based Curriculum Optimization for Neural Machine Translation (2019, background): “Among the very extensive work on handling heterogeneous data in NMT, the closest to ours are techniques that re-weight (Chen et al., 2017) or re-order examples to deal with domain mismatch (van der Wees et al., 2017; Sajjad et al., 2017) or noise (Wang et al., 2018).”
Neural Machine Translation: A Review (2019, background): “[481] showed that a simple concatenation of in-domain and out-domain corpora can already increase the robustness and generalization of NMT significantly.”
Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection (2018, background): “(Sajjad et al., 2017) ﬁnd the usefulness of a similar idea, called model stacking for NMT domain adaptation.”
Unsupervised Domain Clusters in Pretrained Language Models (2020, methodology): “…examples can be used to either (1) train a domain-speciﬁc model from scratch (Axelrod et al., 2011), (2) ﬁne-tune a pre-trained general-domain model (Sajjad et al., 2017; Silva et al., 2018), or (3) prioritize data for annotation as in an Active-Learning framework, if only monolingual data is…”