Larger-Scale Transformers for Multilingual Masked Language Modeling

By Jingfei Du, Myle Ott, Giridhar Anantharaman, Alexis Conneau. 2021

162 citations recorded by the source corpus.

Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger multilingual masked language models, with 3.5B and 10.7B parameters. Our two new models dubbed and outperform XLM-R by 1.8% and 2.4% average accuracy on XNLI. Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0.3% on average while handling 99 more languages. This suggests larger capacity models for language understanding may obtain strong performance on high-resource languages while greatly improving low-resource languages. We make our code and models publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Furu Wei
3. Graham Neubig
4. Mohit Bansal
5. Noah A. Smith
6. Jianfeng Gao
7. Yejin Choi
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Luke S. Zettlemoyer
2. Luc Van Gool
3. Furu Wei
4. Mohit Bansal
5. Dacheng Tao
6. Trevor Darrell
7. Xiaogang Wang
8. Jianfeng Gao

In Large Language Models

1. Luke S. Zettlemoyer
2. Furu Wei
3. Li Dong
4. Yi Tay
5. Myle Ott
6. Omer Levy
7. Shaohan Huang
8. Alexei Baevski

Citations in the corpus

• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “To our knowledge, this is the largest effort to date, in making speech technology accessible for many more languages using publicly available data.”
• XLM-E: Cross-lingual Language Model Pre-training via ELECTRA (2021, background): “Scaling-up model size has shown to improve performance on cross-lingual downstream tasks (Xue et al., 2021; Goyal et al., 2021).”
• Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? (2022, background): “Conneau et al. (2020); Goyal et al. (2021) scaled-up multilingual encoder-only architectures up to 11B parameters while maintaining the original Transformer implementation.”
• Language Models are Few-shot Multilingual Learners (2021, methodology): “Recent advances in pre-trained LMs have been focused on building pre-trained encoders, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019a), ELMO (Peters et al., 2018), ULM-FiT (Howard and Ruder, 2018), ELECTRA (Clark et al., 2019), XLM (Conneau and Lample, 2019), and XLM-R (Conneau et al”
• SwitchPrompt: Learning Domain-Specific Gated Soft Prompts for Classification in Low-Resource Domains (2023, background): “Most LMs which are publicly available have been trained on general-domain corpora (Devlin et al., 2019; Liu et al., 2019; Goyal et al., 2021), such as Wikipedia or the BooksCorpus (Zhu et al., 2015).”
• xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “We train two x COMET versions— x COMET - XL and x COMET - XXL —using the XL (3.5B parameters) and XXL (10.7B parameters) versions of XLM-R (Goyal et al., 2021).”
• Scaling Laws for Generative Mixed-Modal Language Models (2023, background): “Goyal et al. (2021) and Shaham et al. (2022) demonstrated that this interference could occur even on models much smaller than the available training data, but scaling up the model size can improve synergy and alleviate interference.”
• Few-shot Learning with Multilingual Generative Language Models (2021, methodology): “We report machine translation results on popular WMT pairs in Table 11, and a subset of FLORES-101 (Goyal et al., 2021b) in Table 12.”