Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

By Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan. 2020

563 citations recorded by the source corpus.

Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages. Previous work in multilingual pretraining has demonstrated that machine translation systems can be created by finetuning on bitext. In this work, we show that multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one direction, a pretrained model is finetuned on many directions at the same time. Compared to multilingual models trained from scratch, starting from pretrained models incorporates the benefits of large quantities of unlabeled monolingual data, which is particularly important for low resource languages where bitext is not available. We demonstrate that pretrained models can be extended to incorporate additional languages without loss of performance. We double the number of languages in mBART to support multilingual machine translation models of 50 languages. Finally, we create the ML50 benchmark, covering low, mid, and high resource languages, to facilitate reproducible research by standardizing training and evaluation data. On ML50, we demonstrate that multilingual finetuning improves on average 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while improving 9.3 BLEU on average over bilingual baselines from scratch.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution (2021, background): “Tang et al. (2020) was able to extend the multilingual translation model mBART (Liu et al., 2020) from 25 to 50 languages by exploiting the fact that mBART’s vocabulary already supported those additional 25 languages.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “2020), which has achieved state-of-the-art results when finetuning on downstream tasks in NLP (Liu et al., 2020; Devlin et al., 2019; Raffel et al., 2020; Tang et al., 2020).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “, 2015), multilingual machine translation (Johnson et al., 2016; Tang et al., 2020; Fan et al., 2020), and even unsupervised machine translation (Lample et al.”
Investigating Failures of Automatic Translation in the Case of Unambiguous Gender (2021, background): “(iii) mBART-50 is another multilingual model (Tang et al., 2020) that is obtained by “many-to-many” direction fine-tuning of a seed mBART denoising auto-encoder model (Liu et al.”
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, methodology): “We follow the same implementation as Tang et al. (2020) to finetune the mBART model.2 It is noted that mBART has 12 layers with 1024 hidden size, leading to a larger model size than ∆LM.”
Tricks for Training Sparse Translation Models (2021, methodology): “We observe similar trends in ML-50, where adding heating improves low-resource performance by +1.4 BLEU.”
Facebook AI’s WMT20 News Translation Task Submission (2020, background): “Recent work Tang et al. (2020) extended this to multilingual fine-tuning, which can create multilingual translation models from multilingual pre-trained models.”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, methodology): “We used EasyNMT5 to evaluate three machine translation models: mBART50_m2m (Tang et al., 2020; Liu et al., 2020), m2m_100_418M (Fan et al., 2020), and Opus-MT (Tiedemann and Thottingal, 2020), representing the state-of-the-art for publicly available neural machine translations models.”
Models and Datasets for Cross-Lingual Summarisation (2022, methodology): “mBART50 (Tang et al., 2020) is the result of fine-tuning mBART (Liu et al.”
Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “(2020) and the decoder using mBART50, a multilingual denoising auto-encoder pre-trained on 50 languages (Tang et al., 2020).”
Using natural language prompts for machine translation (2022, unknown): “Moreover, learning from multiple tasks has been shown to be helpful for generalization at all stages of training: during pre-training (Aribandi et al., 2021), as a special training stage before ﬁne-tuning (Aghajanyan et al., 2021), and even during ﬁne-tuning (Tang et al., 2020).”
Multilingual training for Software Engineering (2021, background): “Third,we find that quite often a model trained in one programming language achieves surprisingly good performance on a test set in a different language, sometimes even surpassing a model trained on the same language as the test set! Taken together, these findings suggest that pooling data across lan”