Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

By Shijie Wu, Mark Dredze. 2019

753 citations recorded by the source corpus.

Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation (2019, methodology): “Since mBERT is SOTA for zero-shot cross-lingual transfer on POS tagging task (Wu and Dredze, 2019), we also establish state-of-the-art on this dataset by beating mBERT in this setting.”
Unsupervised Cross-lingual Representation Learning at Scale (2019, methodology): “…as seen in a number of benchmarks including cross-lingual natural language inference (Bowman et al., 2015; Williams et al., 2017; Con-neau et al., 2018), question answering (Rajpurkar et al., 2016; Lewis et al., 2019), and named entity recognition (Pires et al., 2019; Wu and Dredze, 2019).”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “Pires et al. (2019) and Wu and Dredze (2019) hypothesize that shared word-pieces help mBERT, based on experiments where the task performance correlated with the amount of shared vocabulary between languages.”
On the Cross-lingual Transferability of Monolingual Representations (2019, background): “Cao et al. (2020) echoed this sentiment, and Wu and Dredze (2019) further observed that mBERT performs better in languages that share many sub-words.”
Towards Debiasing Sentence Representations (2020, background): “More recently, sentence representations such as ELMo [20], BERT [8], and GPT [21] have become the preferred choice for encoding text [30, 17, 31, 23, 26].”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning (2021, background): “By transitioning from English-only to highly multilingual language datasets, models like mT5 [38] and mBERT [37], are an important step for researchers driving globally, equitable availability of information.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, positive): “Multilingual BERT (mBERT; Devlin et al. 2019) is pre-trained with the multilingual masked language modeling (MMLM) task on the monolingual text. mBERT produces cross-lingual representations and performs cross-lingual tasks surprisingly well (Wu and Dredze, 2019).”
Cross-Lingual Natural Language Generation via Pre-Training (2019, background): “Cross-Lingual Pre-Training By pre-training BERT (De-vlin et al. 2019) on corpus of multiple languages, it shows a surprising ability to produce cross-lingual representations (Wu and Dredze 2019).”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, background): “…on large amounts of multilingual data—have been observed to perform surprisingly well when transferring to other languages (Pires et al., 2019; Wu and Dredze, 2019; Wu et al., 2020) and the current state-of-the-art model, XLM-R has been shown to be competitive with the performance of…”
Cross-Lingual Ability of Multilingual BERT: An Empirical Study (2019, background): “Even if we don’t add a language identity marker, BERT learns language identity (Wu & Dredze, 2019).”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Empirical studies shows that ﬁnetuning the ﬁnal layers of BERT account for most of the quality gains on downstream tasks (Kovaleva et al., 2019; Lee et al., 2019).”
CamemBERT: a Tasty French Language Model (2019, background): “We pretrain our model on 256 Nvidia V100 GPUs (32GB each) for 100k steps during 17h. Fine-tuning For each task, we append the relevant predictive layer on top of CamemBERT’s Trans-former architecture.”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “Other work has shown that mBERT outperforms previous state-of-the-art of cross-lingual transfer based on word embeddings on token-level NLP tasks (Wu and Dredze, 2019), and that adding character-level information (Mulcaire et al.”
Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning (2020, methodology): “…2019), pre-trained on extensive monolingual or bilingual resources across numerous languages, have been shown to have surprising cross-lingual adaptation abilities, and ﬁne-tuning them to downstream cross-lingual tasks has achieved promising results (Pires et al., 2019; Wu and Dredze, 2019).”
Investigating Multilingual NMT Representations at Scale (2019, background): “…on tasks including lowresource language translation (Zoph et al., 2016; Nguyen and Chiang, 2017; Neubig and Hu, 2018) and zero or few-shot transfer learning for downstream tasks (Eriguchi et al., 2018; Lample and ∗Work done as part of Google AI Residency Conneau, 2019; Wu and Dredze, 2019).”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, background): “Recent language models (Devlin et al., 2019; Conneau et al., 2020), based on Transformer architectures (Vaswani et al., 2017) and pretrained on massive amounts of multilingual data, have recently surpassed (static) cross-lingual word embedding spaces (Ruder et al., 2019; Glavaš et al., 2019) as the”