mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer

By Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel. 2020

3,278 citations recorded by the source corpus.

The recent “Text-to-Text Transfer Transformer” (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent “accidental translation” in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Graham Neubig
  5. Percy Liang
  6. Yejin Choi
  7. Noah A. Smith
  8. Quoc V. Le

In General Social Sciences

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Furu Wei
  4. Jianfeng Gao
  5. Yejin Choi
  6. Mohit Bansal
  7. Graham Neubig
  8. Yonatan Belinkov

In Large Language Models

  1. Luke S. Zettlemoyer
  2. Hinrich Schütze
  3. Yejin Choi
  4. Mohit Bansal
  5. Jianfeng Gao
  6. Timo Schick
  7. Furu Wei
  8. Hannaneh Hajishirzi

In Natural Language Processing

  1. Graham Neubig
  2. Mohit Bansal
  3. Luke S. Zettlemoyer
  4. Yejin Choi
  5. Jianfeng Gao
  6. Noah A. Smith
  7. Percy Liang
  8. Kyunghyun Cho

Citations in the corpus