Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

By Nora Kassner, Philipp Dufter, Hinrich Schütze. 2021

167 citations recorded by the source corpus.

Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT’s performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Fol-low up studies have introduced improved prompts for eliciting such knowledge (Jiang et al., 2020b) as well as multilingual versions (Jiang et al., 2020a; Kassner et al., 2021).”
Measuring and Improving Consistency in Pretrained Language Models (2021, methodology): “Therefore, following (Xiong et al., 2020; Ravichander et al., 2020; Kassner et al., 2021a), we restrict the PLMs’ output vocabulary to the set of possible gold objects for each relation from the underlining KB.”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, background): “…of multilingual models has mostly focused on general information extraction tasks such as question answering (Clark et al., 2020; Hu et al., 2020; Kassner et al., 2021; Ruder & Sil, 2021) as well as speciﬁc types of reasoning such as commonsense reasoning (Ponti et al., 2020; Lin et al., 2021a)…”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “…et al., 2020; Poerner et al., 2020; Cao et al., 2021) Varying base prompts (Elazar et al., 2021; Heinzerling and Inui, 2021; Jiang et al., 2020a; Kassner et al., 2021) Symbolic rule-based prompting (Kassner et al., 2020; Talmor et al., 2020a) Statement scores (§ 3.2) Single-LM scoring…”
Few-shot Learning with Multilingual Generative Language Models (2021, methodology): “…facilitate cross-lingual transfer, including sub-word tokenizers which enabled efficient, shared vocabulary learning across languages (Kudo and Richard-son, 2018), joint training for efficient knowledge transfer across languages (Pires et al., 2019; Jiang et al., 2020; Kassner et al., 2021), etc.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “XLAMA [252] Factual knowledge Multilingual (53 languages) TQ (Typed Query) Single and Multi token entities TREx [249] and GoogleRE”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, background): “Examples of such recycled resources include the ones introduced by Conneau et al. (2018) and Kassner et al. (2021), among others.”
Few-shot Learning with Multilingual Language Models (2021, methodology): “Jointly training on multiple languages enabled a single model to learn from more data, but prompted further work on investigating the efficiency of knowledge transfer across languages (Pires et al., 2019), (Jiang et al., 2020), (Kassner et al., 2021).”
mGPT: Few-Shot Learners Go Multilingual (2022, methodology): “We probe our models for factual knowledge in 23 languages using the mLAMA dataset (Kassner et al., 2021a).”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, background): “…LMs with respect to paraphrased relations (Elazar et al., 2021a), negation (Ettinger, 2020; Kassner and Schütze, 2020), multilinguality (Jiang et al., 2020; Kassner et al., 2020), singular and plural hypernymy probes (Ravichander et al., 2020) and common-sense constraints (Kassner et al., 2021b).”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “These include differences in factual knowledge (Kassner et al., 2021), grounding of time-of-day expressions (Shwartz, 2022), and social value (Lin et al.”