Cross-lingual Retrieval for Iterative Self-Supervised Training

By Chau Tran, Yuqing Tang, Xian Li, Jiatao Gu. 2020

76 citations recorded by the source corpus.

Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach -- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Discovering Language Model Behaviors with Model-Written Evaluations (2022, methodology): “This method has seen success in machine translation (Schwenk, 2018; Zhang et al., 2020; Tran et al., 2020), text classification (Du et al.”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, methodology): “To mine input output pairs, we follow previous work on unsupervised machine translation and employ the margin function formulation (Artetxe and Schwenk, 2019; Tran et al., 2020) based on K nearest neighbors (KNN).”
One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrieval (2021, methodology): “Tran et al. (2020) introduce CRISS, a self-supervised pre-training approach consisting of a parallel sentence mining module and a sequence-to-sequence model, which are trained iteratively.”
MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark (2020, methodology): “• CRISS (Tran et al., 2020) is pre-trained on parallel data using unsupervised fashion.”
Multilingual Translation from Denoising Pre-Training (2021, background): “In particular, recent works explored pretraining on multilingual unlabeled corpora (Lample and Conneau, 2019; Conneau et al., 2020; Liu et al., 2020; Tran et al., 2020) and significantly improved the”
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning (2020, background): “In particular, recent works explored pre-training on multilingual unlabeled corpus (Lample and Con-neau, 2019; Conneau et al., 2019; Liu et al., 2020; Tran et al., 2020), and signiﬁcantly improved the performance of ﬁne-tuning on machine translation between two languages.”
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, background): “However, the sizes of parallel corpora are limited (Tran et al., 2020), restricting the performance of the cross-lingual language model.”