Are All Languages Created Equal in Multilingual BERT?

By Shijie Wu, Mark Dredze. Workshop on Representation Learning for NLP, 2020

402 citations recorded by the source corpus.

Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit cross-lingual signals. However, these evaluations have focused on cross-lingual transfer with high-resource languages, covering only a third of the languages covered by mBERT. We explore how mBERT performs on a much wider set of languages, focusing on the quality of representation for low-resource languages, measured by within-language performance. We consider three tasks: Named Entity Recognition (99 languages), Part-of-speech Tagging and Dependency Parsing (54 languages each). mBERT does better than or comparable to baselines on high resource languages but does much worse for low resource languages. Furthermore, monolingual BERT models for these languages do even worse. Paired with similar languages, the performance gap between monolingual BERT and mBERT can be narrowed. We find that better models for low resource languages require more efficient pretraining techniques or more data.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/14489ec7893e373a0dcc9555c52b99b2b3a429f6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, neutral): “…tasks such as translation (Jiao et al., 2023; Xu et al., 2024) and addressing low-resource languages (Artetxe and Schwenk, 2019; Wu and Dredze, 2020). mBERT pioneered the multilingual area by demonstrating that training on multilingual data allows different languages to be represented…”
Towards multidomain and multilingual abusive language detection: a survey (2021, background): “Even a study by Wu and Dredze [149] found that 30% of languages in the multilingual BERT model with lower pretraining resources obtain worse performance than without using a pre-trained language model.”
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer (2020, background): “Languages with smallest pretraining corpora are most affected: the largest transfer performance drops are observed with those target languages (Lauscher et al., 2020; Wu and Dredze, 2020).”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “Next to the curse of multilinguality, recent works have shown substantially reduced cross-lingual and monolingual abilities of models for low-resource languages with smaller pre-training data (Wu and Dredze, 2020; Hu et al., 2020; Lauscher et al., 2020b; Artetxe et al., 2020; Pfeiffer et al., 2020b,”
Emergent Structures and Training Dynamics in Large Language Models (2022, background): “Thus, low-379 resource languages often benefit more from the joint training (Wu and Dredze, 2020), while lan-381 guages with abundant resources often achieve bet-382 ter performance when trained on their own (Nozza et al., 2020; Lewis et al., 2020).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “…work which primarily attributes gaps in performance to mBERT being under-trained with respect to individual languages (Rönnqvist et al., 2019; Wu and Dredze, 2020), our results, when disentangling the effect of the tokenizer (as seen in Table 3), convincingly show that a large portion of…”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “, 2020): Research groups are extending NLP research to low-resourced languages (Guzmán et al., 2019; Hu et al., 2020; Wu and Dredze, 2020), and workshops have been established (Haffari et al.”
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “However, to our knowledge, none of existing work has conducted large-scale evaluations of ChatGPT for multiple and diverse languages/tasks as we do.”
Intriguing Properties of Compression on Multilingual Models (2022, methodology): “…including size, family and script, relatedness to other languages, and the inherent difﬁculty of the NER task and the evaluation examples, as studied in related works (e.g., Pires et al., 2019; Wu and Dredze, 2020; Shaffer, 2021; Adelani et al., 2021; Muller et al., 2021; Deshpande et al., 2021).”
Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, background): “Even if more work is needed to establish when such approaches will work well (Marchisio et al., 2020; Artetxe et al., 2020; Wu and Dredze, 2020), having useful monolingual text corpora across languages is clearly a prerequisite to exploring such approaches further.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, background): “Wu and Drezde [144] found that while mBERT does not perform equally well across all 104 languages in its training data, it performed better at NER, POS tagging, and dependency parsing than monolingual models trained with comparable amounts of data for four low-resource languages.”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, neutral): “…similar languages (Conneau et al., 2020; Artetxe et al., 2020), the multilin- በእግዚአብሔር བྷྱཿཤཱནཾིྟ ކަނާތްފަރާތުންނެވެ እግዚአብሔርን བཻཌཱུརྱའི ޖުމްހޫރިއްޔާއެކެވެ ከእግዚአብሔር ེདྷརྨཱཧེཏ ލިބިވަޑައިގެންނެވި የመጀመሪያውን ཝཱཧེཏུནེྟ ބަޔާންކޮށްފައިވަނީ ትምህርታቸውን སྤྲེའུའི އުފަންވެވަޑައިގަ gual models struggle with…”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “The existing work on other tasks and the popularity in other fields like image classification (Xiao et al., 2015; Li et al., 2017; Lee et al., 2018; Mahajan et al., 2018; Li et al., 2020) suggests, however, that distant supervision could be leveraged for more NLP tasks in the future.”
Language Models for Code-switch Detection of te reo Māori and English in a Low-resource Setting (2022, neutral): “Google’s RNNbased GNMT (Google Neural Machine Translation) model (Wu et al., 2016) showed significant improvements in enabling translations to cover many languages, including low-resourced languages.”