mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

By Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang Xian-Ling Mao, Saksham Singhal, Heyan Huang, Furu Wei. 2021

82 citations recorded by the source corpus.

Multilingual T5 pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on seven multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Saksham Singhal
6. Zewen Chi
7. Xia Song
8. Ajit Rajasekharan

In General Social Sciences

1. Li Dong
2. Saksham Singhal
3. Shuming Ma
4. Hany Hassan Awadalla
5. Shaohan Huang
6. Payal Bajaj
7. Dongdong Zhang
8. Zewen Chi

In Large Language Models

1. Furu Wei
2. Li Dong
3. Shaohan Huang
4. Shuming Ma
5. Xia Song
6. Zewen Chi
7. Saksham Singhal
8. Dongdong Zhang

In Natural Language Processing

1. Furu Wei
2. Li Dong
3. Ajit Rajasekharan
4. Sivanesan Sangeetha
5. Katikapalli Subramanyam Kalyan
6. Shaohan Huang
7. Shuming Ma
8. Xia Song

In Machine Translation

1. Li Dong
2. Furu Wei

Citations in the corpus

• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, methodology): “We believe the cross-lingual contrast (Chi et al., 2021b) task explicitly learns the sentence representations, which makes I NFO XLM more effective for the cross-lingual sentence retrieval task.”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “mT6 [91], XLM [63], infoXLM [90], Unicoder [103], ALM [104], XLM-E [89]”
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders (2021, methodology): “Therefore, we follow mT6 (Chi et al., 2021) to improve mT5 with a translation span corruption task.”