How Multilingual is Multilingual BERT?

By Telmo Pires, Eva Schlinger, Dan Garrette. 2019

1,726 citations recorded by the source corpus.

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, neutral): “Similar phenomena have also been reported in other multilingual pre-training approachesin other NLP applications (Pires et al., 2019; Wang et al., 2019; Artetxe et al., 2019).”
Learning Compact Metrics for MT (2021, neutral): “Evidence suggests that training on many languages improves performance on languages for which there is little training data, including the zero-shot setup, in which no ﬁne-tuning data is available (Conneau and Lample, 2019; Sellam et al., 2020b; Conneau et al., 2018; Pires et al., 2019).”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “M-BERT Encoder-decoder Transformer architectures are a very popular choice for text generation.”
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, background): “Subsequently, these contextual methods have been extended to produce multilingual representations by training a single model on text from multiple languages which have proven to be very effective for cross lingual transfer (Wu and Dredze, 2019; Mulcaire et al., 2019; Pires et al., 2019).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “Emergent Cross-Lingual Alignment On understanding cross-lingual alignment from pretrained models, [38] [24] present empirical evidence that there exists cross-lingual alignment structure in the encoder which is trained with multiple languages on a shared masked language modeling task.”
Parallel Corpus Filtering via Pre-trained Language Models (2020, background): “Multilingual BERT (Devlin et al., 2019) projects multilingual sentences into a shared space and has shown a great potential for cross-lingual model transfer (Pires et al., 2019).”
CodeBERT: A Pre-Trained Model for Programming and Natural Languages (2020, methodology): “Training is conducted in a setting similar to that of multilingual BERT (Pires et al., 2019), in which case one pre-trained model is learned for 6 programming languages with no explicit markers used to denote the input programming language.”
Unsupervised Cross-lingual Representation Learning at Scale (2019, methodology): “…as seen in a number of benchmarks including cross-lingual natural language inference (Bowman et al., 2015; Williams et al., 2017; Con-neau et al., 2018), question answering (Rajpurkar et al., 2016; Lewis et al., 2019), and named entity recognition (Pires et al., 2019; Wu and Dredze, 2019).”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, background): “…2018; Peters et al., 2018; Devlin et al., 2019), multilingual extensions of pretrained encoders have recently been shown to be effective for learning deep cross-lingual representations (Eriguchi et al., 2018; Pires et al., 2019; Wu & Dredze, 2019; Lample & Conneau, 2019; Siddhant et al., 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “Languages with a lot of data were subsampled, and some were super-sampled using exponential smoothing. mBERT performs surprisingly well in zero-shot transfer on many tasks (Wu and Dredze, 2019; Pires et al., 2019), although not in language generation (R¨onnqvist et al., 2019).”
The State and Fate of Linguistic Diversity and Inclusion in the NLP World (2020, background): “Also, transfer learning of resourcerich to resource-poor languages have been shown to work better if the respective languages contain similar typological features (Pires et al., 2019).”
On the Cross-lingual Transferability of Monolingual Representations (2019, methodology): “In relation to this, Pires et al. (2019) hypothesized that: ... having word pieces used in all languages (numbers, URLs, etc), which have to be mapped to a shared space forces the co-occurring pieces to also be mapped to a shared space, thus spreading the effect to other word pieces, until…”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Aside from the problem of education and complexity, existing techniques may not apply due to linguistic and morphological differences in the languages, or the scale, domain, or quality of the data (Hu et al., 2020; Pires et al., 2019).”
Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT (2019, background): “Parallel to our work, Pires et al. (2019) shows mBERT has good zero-shot cross-lingual transfer performance on NER and POS tagging.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, background): “…2017) models pretrained on large amounts of multilingual data—have been observed to perform surprisingly well when transferring to other languages (Pires et al., 2019; Wu and Dredze, 2019; Wu et al., 2020) and the current state-of-the-art model, XLM-R has been shown to be competitive with the…”
Cross-Lingual Ability of Multilingual BERT: An Empirical Study (2019, background): “Previous work (Pires et al., 2019) hypothesizes that M-BERT generalizes across languages because these shared word-pieces force the other word-pieces to be mapped to the same shared space.”
Pre-trained models for natural language processing: A survey (2020, background): “Even so, mBERT performs cross-lingual generalization surprisingly well [140].”