Multi-Domain Neural Machine Translation through Unsupervised Adaptation

By M. Amin Farajian, Marco Turchi, Matteo Negri, Marcello Federico. 2017

133 citations recorded by the source corpus.

We investigate the application of Neural Machine Translation (NMT) under the following three conditions posed by real-world application scenarios. First, we operate with an input stream of sentences coming from many different domains and with no predeﬁned order. Second, the sentences are presented without domain information. Third, the input stream should be processed by a single generic NMT model. To tackle the weaknesses of current NMT technology in this unsupervised multi-domain setting, we explore an ef-ﬁcient instance-based adaptation method that, by exploiting the similarity between the training instances and each test sentence, dynamically sets the hyperparameters of the learning algorithm and updates the generic model on-the-ﬂy. The results of our experiments with multi-domain data show that local adaptation outperforms not only the original generic NMT system, but also a strong phrase-based system and even single-domain NMT models speciﬁ-cally optimized on each domain and applicable only by violating two of our afore-mentioned assumptions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “…for high resource languages seems to be dramatically improved (Wu et al., 2016; Hassan et al., 2018), performance on out-of-domain data or low resource languages, still remains pretty poor (Duh et al., 2013; Koehn and Knowles, 2017; Farajian et al., 2017; Dong et al., 2015; Zoph et al., 2016).”
Nearest Neighbor Machine Translation (2020, background): “Finally, various studies have explored retrieving additional information to improve domain adaptation, often using lexicons (Hu et al., 2019), domain-adaptive training (Farajian et al., 2017) or attending over neighbors similar to n -grams in the source (Bapna & Firat, 2019).”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “We share the viewpoint of Farajian et al. (2017), that NMT models ultimately should be able to adapt on-the-fly to on-line streams of diverse data (i.e. language, domain), and thus CL for NMT is essential.”
Non-Parametric Adaptation for Neural Machine Translation (2019, methodology): “NMT has also been shown to perform poorly when dealing with multi-domain data (Farajian et al., 2017a).”
Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem (2020, neutral): “Recent work recognizes that NMT can be adapted to domains with desired attributes using small datasets (Farajian et al., 2017; Michel and Neubig, 2018).”
Guiding Neural Machine Translation with Retrieved Translation Pieces (2018, methodology): “In contrast, Li et al. (2016) and Farajian et al. (2017) use the retrieved sentence pairs to run additional training iterations and ﬁne tune the NMT parameters for each input sentence; Gu et al. (2017) runs the NMT model for each retrieved sentence pair to obtain the NMT encoding and decoding…”
Unsupervised Domain Clusters in Pretrained Language Models (2020, background): “Farajian et al. (2017) explored instance-based data selection in a multi-domain scenario using information retrieval meth-ods.”
Retrieval-Based Neural Code Generation (2018, background): “Recent works on retrieval-guided neural machine translation have been presented by Gu et al. (2018); Amin Farajian et al. (2017); Li et al. (2018); Zhang et al. (2018).”
Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination (2018, background): “To tackle this problem, recently, researchers have carried out many constructive and in-depth studies (Kobus et al., 2016; Zhang et al., 2016; Pryzant et al., 2017; Farajian et al., 2017).”
Sequence to Sequence Mixture Model for Diverse Machine Translation (2018, background): “, 2017), tuning (Luong and Manning, 2015; Farajian et al., 2017) to domain tags (Chu et al.”
A Survey of Domain Adaptation for Neural Machine Translation (2018, methodology): “Farajian et al. (2017) follow the strategy of Li et al. (2016), but propose to dynamically set the hyperparameters (i.e., learning rate and number of epochs) of the learning algorithm based on the similarity of the input sentence and the retrieved sentences for updating the NMT model.”
Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation (2018, methodology): “00001 ) and choose the best result based on the perplexity on the development set, as previous work has suggested that even when using A DAM , continued training can be sensitive to learning rate (Farajian et al., 2017; Li et al., 2018; Kothur et al., 2018).”