Explicit Cross-lingual Pre-training for Unsupervised Machine Translation

By Shuo Ren, Yu Wu, Shujie Liu, M. Zhou, Shuai Ma. 2019

44 citations recorded by the source corpus.

Pre-training has proven to be effective in unsupervised machine translation due to its ability to model deep context information in cross-lingual scenarios. However, the cross-lingual information obtained from shared BPE spaces is inexplicit and limited. In this paper, we propose a novel cross-lingual pre-training method for unsupervised machine translation by incorporating explicit cross-lingual training signals. Specifically, we first calculate cross-lingual n-gram embeddings and infer an n-gram translation table from them. With those n-gram translation pairs, we propose a new pre-training model called Cross-lingual Masked Language Model (CMLM), which randomly chooses source n-grams in the input text stream and predicts their translation candidates at each time step. Experiments show that our method can incorporate beneficial cross-lingual information into pre-trained models. Taking pre-trained CMLM models as the encoder and decoder, we significantly improve the performance of unsupervised machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Direction en-de de-en en-fr fr-en en-ne ne-en en-ro ro-en en-si si-en CMLM [47] 27.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Recent works have shown that applying cross-lingual techniques such as cross-lingual masked LMs could be more efective in initializing the models [157].”
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021, unknown): “The ﬁrst category usually uses pre-trained models as feature extractors of a source language, or initializes the encoder and decoder with pre-trained models separately (Lample and Conneau, 2019; Ren et al., 2019; Yang et al., 2020a; Zhu et al., 2020).”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “[157] further computed cross-lingual n-gram embeddings and derived an n-gram translation table based on them for providing explicit representation learning signals.”
A Multilingual View of Unsupervised Machine Translation (2020, methodology): “…recent work of Artetxe et al. (2019) which leverages tools from statistical MT as well subword information to enrichen their models; (5) the work of Ren et al. (2019) that explicitly attempts to pre-train for UNMT by building cross-lingual n-gram tables and building a new pre-training task based…”