Cross-Lingual Ability of Multilingual BERT: An Empirical Study

By Karthikeyan K, Zihan Wang, Stephen Mayhew, Dan Roth. 2019

386 citations recorded by the source corpus.

Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: this http URL .

Subfields: Artificial Intelligence, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, neutral): “Similar phenomena have also been reported in other multilingual pre-training approachesin other NLP applications (Pires et al., 2019; Wang et al., 2019; Artetxe et al., 2019).”
Of Non-Linearity and Commutativity in BERT (2021, background): “While existing work has focused on analyzing the self-attention heads and the impact of model depth (Karthikeyan et al., 2020; Voita et al., 2019; Michel et al., 2019), we direct our investigation towards the FFN blocks of BERT.”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “To date, the most systematic study of BERT architecture was performed by Wang et al. (2019b).”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, background): “Notice that for the setting without word-piece overlap (Artetxe et al., 2020; Conneau et al., 2020b; K et al., 2020), we hypothesize that the information bottleneck principle (Tishby and Zaslavsky, 2015) tends to transform the cross-lingual structural similarity into isomorphic representations, whic”
Rethinking embedding coupling in pre-trained language models (2020, methodology): “We choose this baseline as its behavior has been thoroughly studied (K et al., 2020; Conneau et al., 2020b; Pires et al., 2019; Wu & Dredze, 2019).”
Emerging Cross-lingual Structure in Pretrained Language Models (2019, background): “Concurrently, Karthikeyan et al. (2020) similarly find anchor points play minor role in learning cross-lingual representation.”
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models (2020, background): “…of these models and what contributes to it (Pires et al., 2019; Wu & Dredze, 2019; Wu et al., 2019; Artetxe et al., 2019; Kudugunta et al., 2019; Karthikeyan et al., 2020), the limitation of such training paradigm (Arivazhagan et al., 2019; Wang et al., 2020b), and how to further improve it by…”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, background): “It is worth pointing out that K et al. (2020) disagree and claim that subword overlap is not as important for transfer.”
Ranking Transfer Languages with Pragmatically-Motivated Features for Multilingual Sentiment Analysis (2020, background): “sequent studies have examined the cross-lingual ability of these models (Wu and Dredze, 2019; Pires et al., 2019; K et al., 2019).”
Emergent Structures and Training Dynamics in Large Language Models (2022, background): “Interestingly, Conneau et al. (2020b) and K et al. (2020) showed that the universal rep-361 The size of each language's corpus in the lan-377 guage model's training set has been shown to be decisive for transfer to that language.”
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, background): “, 2021c), but also show an impressive cross-lingual transferability (Wu and Dredze, 2019; K et al., 2020; Hu et al., 2020b; Chi et al., 2021a).”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, background): “This is an expected outcome since cross-lingual effectiveness depends explicitly on model size (K et al., 2020).”
Explicit Alignment Objectives for Multilingual Bidirectional Encoders (2020, background): “These methods can only implicitly learn which words and structures correspond to each-other across languages in an entirely unsupervised fashion, but are nonetheless quite effective empirically (Conneau et al., 2020b; K et al., 2020).”