X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models

By Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, Graham Neubig. 2020

165 citations recorded by the source corpus.

Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as "Punta Cana is located in _." However, while knowledge is both written and queried in many languages, studies on LMs' factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for 23 typologically diverse languages. To properly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algorithms to generate multi-token predictions. Extensive experimental results provide insights about how well (or poorly) current state-of-the-art LMs perform at this task in languages with more or fewer available resources. We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages. Benchmark data and code have been released at https://x-factr.github.io.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Fol-low up studies have introduced improved prompts for eliciting such knowledge (Jiang et al., 2020b) as well as multilingual versions (Jiang et al., 2020a; Kassner et al., 2021).”
GPT Understands, Too (2021, methodology): “In light of the challenge, recent works propose to automate the search procedure of discrete prompts by mining the training corpus (Jiang et al., 2020b), gradient-based searching (Shin et al., 2020), and us-ing pretrained generative models (Gao et al., 2020).”
Knowledge Neurons in Pretrained Transformers (2021, methodology): “Third, we use the single-word blank in cloze queries for simplicity, which requires multi-word extensions (Jiang et al., 2020a).”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “For the latter, initial results on prompts beyond English indicate high variability in pretrained LM performance across languages and poor performance on prompts with multi-token answers (Jiang et al., 2020a; Kassner et al., 2021).”
Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022, background): “Previous works have shown that large PLMs contain a signiﬁcant amount of knowledge, which can be recovered by querying PLMs with appropriate prompts (Petroni et al., 2019; Bouraoui et al., 2020; Jiang et al., 2020a,b; Wang et al., 2020).”
A Review on Language Models as Knowledge Bases (2022, background): “They also lack social intelligence (Liang et al., 2021), an understanding of time (Dhingra et al., 2021; Lazaridou et al., 2021), causality (Li et al., 2021), dealing with uncommon facts (Poerner et al., 2020; Jiang et al., 2020a) and counterfactual reasoning (Feng et al., 2021).”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, unknown): “Contemporaneous work by Jiang et al. (2020a) provides manual translations of LAMA templates for 23 languages respecting grammatical gender and inﬂection constraints.”
Few-shot Learning with Multilingual Generative Language Models (2021, background): “…facilitate cross-lingual transfer, including sub-word tokenizers which enabled efficient, shared vocabulary learning across languages (Kudo and Richard-son, 2018), joint training for efficient knowledge transfer across languages (Pires et al., 2019; Jiang et al., 2020; Kassner et al., 2021), etc.”
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models (2022, unknown): “We take inspiration from [38] and performs zero-shot VideoQA inference by using multiple mask tokens decoded in parallel.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “X-FACTR [257] Factual knowledge Multilingual (26 languages) UnTQ Single and Multi-token entities TREx [249]”
Few-shot Learning with Multilingual Language Models (2021, background): “Jointly training on multiple languages enabled a single model to learn from more data, but prompted further work on investigating the efficiency of knowledge transfer across languages (Pires et al., 2019), (Jiang et al., 2020), (Kassner et al., 2021).”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Pre-training data embodies a wealth of real-world factual knowledge, enabling LLMs to capture and subsequently encode vast of factual knowledge within their parameters (Petroni et al., 2019; Jiang et al., 2020; Roberts et al., 2020).”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, unknown): “However, as this process was relatively laborintensive, we recommend the usage of morphologically aware templates similar to Jiang et al. (2020) for future work.”
MTG: A Benchmark Suite for Multilingual Text Generation (2021, background): “A significant body of works have been committed to the construction of multilingual datasets covering diverse tasks (Hu et al., 2020; Jiang et al., 2020; Longpre et al., 2020).”
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021, background): “In general, multi-word targets are difﬁcult to assess as PPL is not well-deﬁned for masked language models like BERT (Jiang et al., 2020a; Salazar et al., 2020).”
Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions (2022, background): “, 2021a), negation (Ettinger, 2020; Kassner and Schütze, 2020), multilinguality (Jiang et al., 2020; Kassner et al., 2020), singular and plural hypernymy probes (Ravichander et al.”
Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies (2021, background): “Building on top of the BERT’s masking strategy (Devlin et al., 2019), a wide variety of approaches has been proposed (Liu et al., 2019; Yang et al., 2019; Jiang et al., 2020).”
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2020, background): “However, at the same time, LMs are obviously not omnipotent, and still fail to provide appropriate answers in many cases, such as when dealing with uncommon facts (Poerner et al., 2019; Jiang et al., 2020a) or complex reasoning (Talmor et al., 2019a).”