Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary

By Surafel Melaku Lakew, Aliia Erofeeva, Matteo Negri, Marcello Federico, Marco Turchi. 2018

72 citations recorded by the source corpus.

We propose a method to transfer knowledge across neural machine translation (NMT) models by means of a shared dynamic vocabulary. Our approach allows to extend an initial model for a given language pair to cover new languages by adapting its vocabulary as long as new data become available (i.e., introducing new vocabulary items if they are not included in the initial model). The parameter transfer mechanism is evaluated in two scenarios: i) to adapt a trained single language NMT system to work with a new language pair and ii) to continuously add new language pairs to grow to a multilingual NMT system. In both the scenarios our goal is to improve the translation performance, while minimizing the training convergence time. Preliminary experiments spanning five languages with different training data sizes (i.e., 5k and 50k parallel sentences) show a significant performance gain ranging from +3.85 up to +13.63 BLEU in different language directions. Moreover, when compared with training an NMT model from scratch, our transfer-learning approach allows us to reach higher performance after training up to 4% of the total training steps.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Machine Translation

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…(Nichol et al., 2018; Baydin et al., 2017), model parameters (Ha et al., 2016a; Platanios et al., 2018) and models that learn new tasks with high sample ef-ﬁciency (Finn et al., 2017) without forgetting existing tasks or languages (Rusu et al., 2016; Kirk-patrick et al., 2017; Lakew et al., 2018).”
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “Other works (Zoph et al., 2016; Lakew et al., 2018, 2019; Escolano et al., 2019) have studied repurposing translation models as initializations for bilingual models for a target low-resource language pair.”
Adapting Multilingual Neural Machine Translation to Unseen Languages (2019, methodology): “Adaptation of Pre-Trained Model: We experiment with the adaptation of the multilingual NMT system by preserving the initial model vocabulary (DirAdapt) or dynamically updating it to include new items (DynAdapt), as in [12].”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “Lakew et al. (2018b) dynamically updated the vocabulary of the parent model with the low-resource language pair before transferring parameters.”
Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders (2020, background): “(Lakew et al., 2018) proposes a model based on the addition of new languages to an already trained system by vocabulary adaptation and transfer learning.”
From Bilingual to Multilingual Neural Machine Translation by Incremental Training (2019, background): “Recent work proposes a new approach to add new languages to a system by adapting the vocabulary (Lakew et al., 2018), relying on the shared tokens between the languages to share model parameters, showing that the amount of shared tokens between the languages had an impact in the model performance.”
The University of Maryland’s Kazakh-English Neural Machine Translation System at WMT19 (2019, background): “While much work addresses this problem via semi-supervised learning from monolingual text (Sennrich et al., 2016; He et al., 2016), we focus on transfer learning from another language pair (Zoph et al., 2016; Nguyen and Chiang, 2017; Lakew et al., 2018).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “All the LRL-NMT techniques (data augmentation, unsupervised NMT, semi-supervised NMT, multi-NMT, and transfer learning for NMT) except pivoting show an upward trend with respect to the research publications, suggesting that these techniques have established themselves as the de facto solutions for L”
Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT (2020, methodology): “Lakew et al. (2018) transfer a source NMT model to a target NMT model (e.g. De-En to Nl-En).”
Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation (2019, background): “…for transferring proper parameters (Zoph et al., 2016; Gu et al., 2018b), lexical (Zoph et al., 2016; Nguyen and Chiang, 2017; Gu et al., 2018a; Lakew et al., 2018), and syntactic (Gu et al., 2018a; Murthy et al., 2018) knowledge from a resource-rich language pair to a resource-poor language…”