Unsupervised Cross-lingual Representation Learning at Scale
By Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco (Paco) Guzmán, Edouard Grave, Myle Ott, Luke S. Zettlemoyer, Veselin Stoyanov. 2019
8,807 citations recorded by the source corpus.
This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code and models publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Graham Neubig
- Percy Liang
- Yejin Choi
- Noah A. Smith
- Furu Wei
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Mohit Bansal
- Andrew Zisserman
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Yejin Choi
- Jianfeng Gao
- Noah A. Smith
- Jason Aaron Edward Weston
- Percy Liang
In Representation & Self-Supervised Learning
- Ivan Vulic
- Iryna Gurevych
- Graham Neubig
- Sebastian Ruder
- Hinrich Schütze
- Alexis Conneau
- Furu Wei
- John Wieting
Citations in the corpus
- Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “Scaling-up the training data and model parameters has been a key factor in pre-training (Devlin et al., 2019; Conneau et al., 2019; Raffel et al., 2019).”
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, methodology): “, 2019) as language model; for x-stance, we use XLM-R (Conneau et al., 2019).”
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “fer and uses multilingual pre-trained models such as XLM-R (Conneau et al., 2020).”
- Learning Compact Metrics for MT (2021, neutral): “The phenomenon has been well-studied for tasks such as translation (Aharoni et al., 2019) and language inference (Conneau et al., 2020b), but it is less well understood in the context of evaluation metrics.”
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “3 for XLM-R (Conneau et al., 2019), and α = 0 .”
- Beyond English-Centric Multilingual Machine Translation (2020, background): “These corpora are limited in size and domain, and an alternative is to mine parallel data (Resnik, 1999; Utiyama and Isahara, 2003) in large collections of monolingual data (Conneau et al., 2019; Wenzek et al., 2019).”
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Similarly in natural language processing, scaling Transformers [10] yielded consistent gains on language understanding tasks [4, 11, 12], cross-lingual down-stream transfer [4, 13] and (massively-)multilingual neural machine translation [14, 15, 16].”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We conducted validation experiments using 150k and 250k vocabulary sizes to make comparisons with existing multilingual modeling literature easier (Conneau et al., 2020; Xue et al., 2021).”
- Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “LM refers to Facebook’s Transformer-based Universal Language Model for text translation [25].”
- Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “We use the same data used to train the BASE language model (Lewis et al., 2021), which consists of approximately 100B to-kens, combining corpora used in RoBERTa (Liu et al., 2019) with the English subset of the CC100 corpus (Conneau et al., 2020).”
- Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “XLM-R) without contrastive pre-training, we use a temperature τ of 1 (resp.”
- BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “MLM is also successfully applied to pretrain multilingual language representations (Conneau and Lample, 2019; Conneau et al., 2020; Chi et al., 2021b).”
- When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “Existing dialogue models are primarily trained on human-human conversations (Conneau et al., 2019; Baumgartner et al., 2020; Smith et al., 2020).”
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Thirdly, we find a minimal and inconsistent “curse of multilinguality” (Conneau et al., 2020; Pfeiffer et al., 2022) for BLOOM.”
- Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “We also conduct experiments with multilingual BERT and XLMRoberta (Conneau et al., 2020).”
- BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, methodology): “, 2019) with the English subset of the CC100 corpus (Conneau et al., 2020).”
- HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, background): “One by-product of this type of filtering is that it also produced high-quality documents by default 3 ; thus, we opted out of model-based filter-ing of documents such as CC-100 (Conneau et al., 2019).”