Multilingual Translation from Denoising Pre-Training

By Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Vishrav Chaudhary, Jiatao Gu, Angela Fan. 2021

146 citations recorded by the source corpus.

Recent work demonstrates the potential of training one model for multilingual machine translation. In parallel, denoising pretraining using unlabeled monolingual data as a starting point for finetuning bitext machine translation systems has demonstrated strong performance gains. However, little has been explored on the potential to combine denoising pretraining with multilingual machine translation in a single model. In this work, we fill this gap by studying how multilingual translation models can be created through multilingual finetuning . Fintuning multilingual model from a denoising pretrained model incorporates the benefits of large quantities of unlabeled monolingual data, which is particularly important for low resource languages where bitext is rare. Further, we create the ML50 benchmark to facilitate re-producible research by standardizing training and evaluation data. On ML50, we show that multilingual finetuning significantly improves over multilingual models trained from scratch and bilingual finetuning for translation into English. We also find that multilingual fine-tuning can significantly improve over multilingual models trained from scratch for zero-shot translation on non-English directions. Finally, we discuss that the pretraining and finetuning paradigm alone is not enough to address the challenges of multilingual models for to-Many directions performance.

Subfields: Artificial Intelligence, Large Language Models, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Orhan Firat
2. Ankur Bapna
3. Graham Neubig
4. Markus Freitag
5. Quoc V. Le
6. Yonghui Wu
7. Felix Stahlberg
8. Mohit Bansal

In Large Language Models

1. Ankur Bapna
2. Jian-Yun Nie
3. Xavier García
4. Yuan Cao
5. Ji-rong Wen
6. Alon Lavie
7. Tianyi Tang
8. Junyi Li

In Machine Translation

1. Markus Freitag
2. Nitika Mathur
3. Ricardo Rei
4. Alon Lavie
5. Ankur Bapna
6. Craig Alan Stewart
7. Orhan Firat
8. Isaac Caswell

Citations in the corpus

• MAESTRO: Matched Speech Text Representations through Modality Matching (2022, neutral): “Finetune: ST-only; mBART decoder init [40]”
• Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “Models such as MBART50 (Tang et al., 2021) and M2M100 (Fan et al., 2021) are publicly available, easy to use and have recently been used as baselines and/or as a backbone for new research.”
• Pre-Trained Language Models for Text Generation: A Survey (2022, background): “[178] proposed to fine-tune mBART on multiple language pairs, which is called multilingual fine-tuning.”
• Building Machine Translation Systems for the Next Thousand Languages (2022, background): “, 2019) and massively multilingual modeling (Firat et al., 2016a; Johnson et al., 2017; Aharoni et al., 2019a; Arivazhagan et al., 2019; Tang et al., 2021; Fan et al., 2021) have enabled high quality machine translation systems that can support over 100 languages.”