Unsupervised Cross-lingual Representation Learning at Scale

By Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco (Paco) Guzmán, Edouard Grave, Myle Ott, Luke S. Zettlemoyer, Veselin Stoyanov. 2019

8,807 citations recorded by the source corpus.

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code and models publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Graham Neubig
  5. Percy Liang
  6. Yejin Choi
  7. Noah A. Smith
  8. Furu Wei

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Mohit Bansal
  8. Andrew Zisserman

In Natural Language Processing

  1. Mohit Bansal
  2. Graham Neubig
  3. Luke S. Zettlemoyer
  4. Yejin Choi
  5. Jianfeng Gao
  6. Noah A. Smith
  7. Jason Aaron Edward Weston
  8. Percy Liang

In Representation & Self-Supervised Learning

  1. Ivan Vulic
  2. Iryna Gurevych
  3. Graham Neubig
  4. Sebastian Ruder
  5. Hinrich Schütze
  6. Alexis Conneau
  7. Furu Wei
  8. John Wieting

Citations in the corpus