Exploiting Multilingualism through Multistage Fine-Tuning for Low-Resource Neural Machine Translation

By Raj Dabre, Atsushi Fujita. 2019

63 citations recorded by the source corpus.

This paper highlights the impressive utility of multi-parallel corpora for transfer learning in a one-to-many low-resource neural machine translation (NMT) setting. We report on a systematic comparison of multistage fine-tuning configurations, consisting of (1) pre-training on an external large (209k–440k) parallel corpus for English and a helping target language, (2) mixed pre-training or fine-tuning on a mixture of the external and low-resource (18k) target parallel corpora, and (3) pure fine-tuning on the target parallel corpora. Our experiments confirm that multi-parallel corpora are extremely useful despite their scarcity and content-wise redundancy thus exhibiting the true power of multilingualism. Even when the helping target language is not one of the target languages of our concern, our multistage fine-tuning can give 3–9 BLEU score gains over a simple one-to-one model.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Complete Multilingual Neural Machine Translation (2020, neutral): “But there exist several approaches that make use of the multi-view structure in the data, such as Dabre et al. (2019), who explored the use of small multi-parallel corpora a for one-to-many NMT.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, methodology): “All the LRL-NMT techniques (data augmentation, unsupervised NMT, semi-supervised NMT, multi-NMT, and transfer learning for NMT) except pivoting show an upward trend with respect to the research publications, suggesting that these techniques have established themselves as the de facto solutions for L”
Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment (2023, background): “However, these PLMs may not perform optimally when applied to specific tasks like sentiment analysis, question answering, or translation due to a lack of appropriate domain knowledge [69], [70], [71].”
Softmax Tempering for Training Neural Machine Translation Models (2020, methodology): “In the literature, multilingualism has been successfully lever-aged for improving translation quality (Firat et al., 2016; Zoph et al., 2016; Dabre et al., 2019), and our method could possibly complement the impact of multilingualism due to its data and language independent nature.”
Multilingual Translation from Denoising Pre-Training (2021, background): “Bitext data pretraining and finetuning aiming at creating multiple machine translation models for different translation directions has also be explored (Dabre et al., 2019; Lin et al., 2020).”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “Fine-tuning is beneficial in very low-resource scenarios [37], but gains may be limited due to catastrophic forgetting.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “DAIFT is commonly used in machine translation to eliminate the issue of unseen languages in translation pairs [39, 154].”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, background): “Multilingual parallel corpora are even more scarce for South Asian languages, but exploiting multi-parallel corpora in a one-to-many low resource setting yields considerable gains in BLEU score (Dabre et al., 2019).”