nmT5 - Is parallel data still relevant for pre-training massively multilingual language models?
By Mihir Kale, Aditya Siddhant, Noah Constant, Melvin Johnson, Rami Al-Rfou, Linting Xue. 2021
28 citations recorded by the source corpus.
Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks. In this paper, we investigate the impact of incorporating parallel data into mT5 pre-training. We find that multi-tasking language modeling with objectives such as machine translation during pre-training is a straightforward way to improve performance on downstream multilingual and cross-lingual tasks. However, the gains start to diminish as the model capacity increases, suggesting that parallel data might not be as essential for larger models. At the same time, even at larger model sizes, we find that pre-training with parallel data still provides benefits in the limited labelled data regime
Subfields: Artificial Intelligence, Large Language Models, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Large Language Models
In Machine Translation
Citations in the corpus
- mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Past work has also leveraged parallel data to improve multilingual text representations, e.g. with TLM (Conneau & Lample, 2019), explicit alignment (Hu et al., 2021) or nmT5 (Kale et al., 2021).”