mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
By Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel. 2020
3,278 citations recorded by the source corpus.
The recent “Text-to-Text Transfer Transformer” (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent “accidental translation” in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.
Subfields: Artificial Intelligence, General Social Sciences, Large Language Models, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Yejin Choi
- Noah A. Smith
- Quoc V. Le
In General Social Sciences
- Luke S. Zettlemoyer
- Hinrich Schütze
- Furu Wei
- Jianfeng Gao
- Yejin Choi
- Mohit Bansal
- Graham Neubig
- Yonatan Belinkov
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Mohit Bansal
- Jianfeng Gao
- Timo Schick
- Furu Wei
- Hannaneh Hajishirzi
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Noah A. Smith
- Percy Liang
- Kyunghyun Cho
Citations in the corpus
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “In comparison, mT5 (Xue et al., 2021b) and ByT5 (Xue et al., 2021a) were trained on 6x and 1.5x as much non-English text as PaLM, respectively.”
- Scaling Up Models and Data with t5x and seqio (2022, background): “• Scalable T5 • mT5 (Xue et al., 2021) • ByT5 (Xue et al., 2022) These model implementations are minimal in the sense that they only use Flax with limited abstractions as opposed to using higher-level libraries built on top of Flax (e.g. Flaxformer).”
- Learning Compact Metrics for MT (2021, neutral): “The setup for the smaller models is similar to Chung et al. (2021), except that RemBERT uses on mC4 (Xue et al., 2020) and Wikipedia while we use Wikipedia only.”
- Deduplicating Training Data Makes Language Models Better (2021, background): “These datasets have grown from single gigabytes to as much as a terabyte over the past few years (Chelba et al., 2013; Xue et al., 2020; Graff et al., 2003; Brown et al., 2020).”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We conducted validation experiments using 150k and 250k vocabulary sizes to make comparisons with existing multilingual modeling literature easier (Conneau et al., 2020; Xue et al., 2021).”
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “To the best of our knowledge, GPT-NeoX-20B is the largest dense language model to ever be publicly released with a several-way tie for second place at 13 billion parameters (Artetxe et al., 2021; Xue et al., 2020, 2021) and many more models at the 10-11B parameter scale.”
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, methodology): “In the same manner, we train the multilingual variant mTk-INSTRUCT based on the mT5 model (Xue et al., 2021).”
- FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, methodology): “We explore fine-tuning from a multilingual model which has been pre-trained with the same speech data, and 10TiB of unlabeled text from mC4 corpus which includes 101 languages [26].”
- MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “We use three different setups to study the effectiveness of unlabeled text corpora: (1) VoxPopuli text dataset (VP-T, 3GB) (2) mC4 [2] spanning 101 languages (15TB) and (3) VP-T and mC4.”
- mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “We evaluate our models under both the zero-shot and translate-train-all settings (Ruder et al., 2021), and compare performance against mT5 (Xue et al., 2021a).”
- XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, methodology): “In natural language processing, language models are trained on very large datasets, spanning billions of documents such as CC100 [15] or mC4 [9] to fit models with tens of billions and even trillions of parameters [16] with strong results on established benchmarks.”
- LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “We consider four possible transformer-based text models [62]—the transformer from ViT-B [20] which also resembles that used in CLIP [45], T5-base [46], mT5-base [66], and the classic BERT-base [16]—and whether to initialise them randomly, or from a pre-trained checkpoint.”