ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

By Ouyang Xuan, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang. 2020

121 citations recorded by the source corpus.

Recent studies have demonstrated that pre-trained cross-lingual models achieve impressive performance in downstream cross-lingual tasks. This improvement benefits from learning a large amount of monolingual and parallel corpora. Although it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for low-resource languages. In this paper, we propose Ernie-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to overcome the constraint that the parallel corpus size places on the model performance. Our key insight is to integrate back-translation into the pre-training process. We generate pseudo-parallel sentence pairs on a monolingual corpus to enable the learning of semantic alignments between different languages, thereby enhancing the semantic modeling of cross-lingual models. Experimental results show that Ernie-M outperforms existing cross-lingual models and delivers new state-of-the-art results in various cross-lingual downstream tasks. The codes and pre-trained models will be made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Zhe Gan
3. Jianfeng Gao
4. Graham Neubig
5. Furu Wei
6. Jingjing Liu
7. Zhiyuan Liu
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Zhe Gan
2. Mohit Bansal
3. Luc Van Gool
4. Dacheng Tao
5. Cordelia Schmid
6. Jianfeng Gao
7. Trevor Darrell
8. Andrew Zisserman

In Natural Language Processing

1. Graham Neubig
2. Mohit Bansal
3. Xipeng Qiu
4. Pengfei Liu
5. Furu Wei
6. Jianfeng Gao
7. Zhe Gan
8. Sebastian Ruder

In Representation & Self-Supervised Learning

1. Zhe Gan
2. Linjie Li
3. Jingjing Liu
4. Lijuan Wang
5. Zicheng Liu
6. Mohit Bansal
7. Furu Wei
8. Graham Neubig

In Machine Translation

1. Graham Neubig
2. Mohit Bansal
3. Li Dong
4. Melvin Johnson
5. Orhan Firat
6. Mihir Kale
7. Sebastian Ruder
8. Yang Liu

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “ERNIE-M (Ouyang et al., 2020) proposes back-translation masked language modeling (BTMLM), and expands the scale of parallel corpora through back-translation mechanisms.”
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, background): “Publicly accessible large-scale multi-task benchmarks [12, 62, 61, 23, 37] have facilitated recent advances [16, 72, 42, 71, 64, 18, 47] in NLP.”
• mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “VECO (Luo et al., 2020) pretrains a variable cross-lingual pre-training model that learns unified language representations for both NLU and NLG. ERNIE-M (Ouyang et al., 2020) utilizes the back-translation masked language modeling task that generates pseudo parallel sentence pairs for learning TLM. E”
• nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? (2021, methodology): “, 2020) could be improved further by using parallel data (Conneau and Lample, 2019; Hu et al., 2020a; Ouyang et al., 2020; Luo et al., 2020).”
• XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, methodology): “ERNIE-M (Ouyang et al., 2020) incorporates back-translation into language modeling.7”