Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries

By Benjamin Heinzerling, Kentaro Inui. 2020

156 citations recorded by the source corpus.

Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose name is found in common LM vocabularies. Furthermore, a major benefit of this paradigm, i.e., querying the KB using natural language paraphrases, is underexplored. Here we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to handle millions of entities and present a detailed case study on paraphrased querying of facts stored in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.

Subfields: Artificial Intelligence, Management Science and Operations Research, Large Language Models, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Large Language Models

In Knowledge Graphs & Representation

Citations in the corpus

Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, background): “The ﬁrst is that this is a useful way to understand and speak about how LMs behave.”
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021, methodology): “Prior efforts have also applied template and probe-based methods (Bouraoui et al., 2020; Petroni et al., 2019; Jiang et al., 2020b; Roberts et al., 2020; Heinzerling and Inui, 2020) to extract relational knowledge from large pretrained models; we draw upon these techniques in this work.”
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models (2021, unknown): “…aspects, such as negation (Kassner and Sch¨utze, 2020), easy to guess names (Poerner et al., 2020), integrating adapters (Wang et al., 2020) or ﬁnding alternatives to a “ﬁll-in-the-blank” approach with single-token answers (Bouraoui et al., 2020; Heinzerling and Inui, 2020; Jiang et al., 2020b).”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “For the former, it has been found that pretrained BERT-based LMs typically do not output consistent answers for prompt paraphrases, although their consistency can again be greatly improved by targeted pretraining (Elazar et al., 2021; Heinzerling and Inui, 2021).”
When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [38], [39], [40], [41], [42], [43], [40], [41], [44], [45], [46] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
Explaining Toxic Text via Knowledge Enhanced Text Generation (2022, background): “Prior work such as Heinzerling and Inui (2021), argue that large pretrained models can in fact serve as knowledge bases.”