Finding Universal Grammatical Relations in Multilingual BERT

By Ethan A. Chi, John Hewitt, Christopher D. Manning. 2020

176 citations recorded by the source corpus.

Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations are shared cross-lingually. To better understand this overlap, we extend recent work on finding syntactic trees in neural networks’ internal representations to the multilingual setting. We show that subspaces of mBERT representations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representations of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evidence suggests that even without explicit supervision, multilingual masked language models learn certain linguistic universals.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021, background): “, 2019b) or full dependency parses (Hewitt and Manning, 2019; Chi et al., 2020) as a proxy for discovering latent syntactic knowledge within the model.”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “…2020) perform on linguistic tasks (Goldberg, 2019; Hewitt and Man-ning, 2019; Tenney et al., 2019; Elazar et al., 2021), commonsense (Forbes et al., 2019; Da and Kasai, 2019; Zhang et al., 2020) and reasoning (Talmor et al., 2020; Kassner et al., 2020), usually assessed by measures of accuracy.”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “For instance, consistent dependency trees can be learnt from the representations of intermediate layers indicating syntactic abstractions in mBERT (Limisiewicz et al., 2020; Chi et al., 2020a).”
Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation (2022, background): “Chi et al. (2020) prove that mBERT has a joint subspace of universal syntactic relations.”
An Information Theoretic View on Selecting Linguistic Probes (2020, background): “Recently, various probing tasks have queried the representations of, e.g., contextualized word embeddings (Tenney et al., 2019a,b) and sentence embeddings (Linzen et al., 2016; Chen et al., 2019; Alt et al., 2020; Kassner and Sch¨utze, 2020; Maudslay et al., 2020; Chi et al., 2020).”
Emergent Structures and Training Dynamics in Large Language Models (2022, background): “Chi et al. (2020) high-175 lighted syntactic generalizations in multilingual language models via structured probing, and ¸Sahin et al. (2020) propose a framework for multilingual morpho-syntactic probing, with 15 probing tasks for multiple languages, showing that, while cross-180 lingual…”
Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation (2020, background): “While existing body of work (Lin et al., 2019; Tenney et al., 2019; Liu et al., 2019a; Kulmizev et al., 2020; Chi et al., 2020) probes transformers for structural phenomena, our work is more pragmatically motivated.”
Models and Datasets for Cross-Lingual Summarisation (2022, methodology): “In the zero-shot scenario, a monolingual sum-mariser English summariser is used for cross-lingual summarisation and we assume that the parameters of the English model will be shared to a certain extent across languages (Chi et al., 2020a).”
Probing Pretrained Language Models for Lexical Semantics (2020, background): “…structures (Liu et al., 2019a; Belinkov and Glass, 2019; Tenney et al., 2019); e.g., they have been extensively probed for syntactic knowledge (Hewitt and Manning, 2019; Jawahar et al., 2019; Kulmizev et al., 2020; Chi et al., 2020, inter alia) and morphology (Edmiston, 2020; Hofmann et al., 2020).”
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models (2021, background): “…why pre-trained language models perform well on downstream tasks either in monolingual (Hewitt and Manning, 2019; Jawahar et al., 2019; Liu et al., 2019a; Tenney et al., 2019a,b) or multi-lingual settings (Wu and Dredze, 2019; Chi et al., 2020; Kulmizev et al., 2020; Vuli´c et al., 2020).”
Large Language Models (2023, methodology): “This was done for BERT in [27, 56, 88] by the following procedure: denote the embedding (in a fixed layer) of word i as u i , then the model learns a projection P on this space, such that the distances d ( i, j ) ≡ || P ( u i − u j ) || in this inner product well approximate the distance be-tween…”
Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020, background): “Furthermore, concurrent work (Chi et al., 2020) has already found qualitative evidence of syntactic dependency types emergent in the special case of multilingual structural probes (Hewitt and Man-ning, 2019).”
Intriguing Properties of Compression on Multilingual Models (2022, background): “Model We use the cased multilingual BERT (mBERT) (Devlin et al., 2019) for all our experiments because it is one of the most widely used and studied multilingual LMs (e.g., Pires et al., 2019; Rönnqvist et al., 2019; Wu and Dredze, 2019; Wang et al., 2020; Chi et al., 2020 trained on Wikipedia data ”