Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information

By Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, Lei Li. 2020

134 citations recorded by the source corpus.

We investigate the following question for machine translation (MT): can we develop a single universal MT model to serve as the common seed and obtain derivative and improved models on arbitrary language pairs? We propose mRASP, an approach to pre-train a universal multilingual neural machine translation model. Our key idea in mRASP is its novel technique of random aligned substitution, which brings words and phrases with similar meanings across multiple languages closer in the representation space. We pre-train a mRASP model on 32 language pairs jointly with only public datasets. The model is then fine-tuned on downstream language pairs to obtain specialized MT models. We carry out extensive experiments on 42 translation directions across a diverse settings, including low, medium, rich resource, and as well as transferring to exotic language pairs. Experimental results demonstrate that mRASP achieves significant performance improvement compared to directly training on those target pairs. It is the first time to verify that multiple low-resource language pairs can be utilized to improve rich resource MT. Surprisingly, mRASP is even able to improve the translation quality on exotic languages that never occur in the pre-training corpus. Code, data, and pre-trained models are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Do Language Models Plagiarize? (2022, methodology): “Such text alignment algorithms are applicable to various tasks such as text-reuse detection [51] and translation alignment [33].”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “OrderPreservation CSP [199]; mRASP [116]; Wada et al.”
LightSeq: A High Performance Inference Library for Transformers (2021, methodology): “In recent years, with the introduction of Transformer model (Vaswani et al., 2017b), many pre-trained language models such as BERT, GPT, and mRASP have also been widely used in these tasks (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2020; Lin et al., 2020).”
Multilingual Translation from Denoising Pre-Training (2021, background): “Bitext data pretraining and finetuning aiming at creating multiple machine translation models for different translation directions has also be explored (Dabre et al., 2019; Lin et al., 2020).”
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021, background): “There is also some work trying to explicitly introduce cross-lingual information in a code-switch way during the sequence-to-sequence pre-training, such as CSP (Yang et al., 2020b) and mRASP (Lin et al., 2020).”
Counter-Interference Adapter for Multilingual Machine Translation (2021, methodology): “To study whether the embedding adapter alleviates multilingual embedding conflation, we calculate the Average Cosine Similarity(ACS) (Lin et al., 2020) of words with the same meaning across different languages to verify if embedding adapter help to align cross-lingual synonyms.”