Relational World Knowledge Representation in Contextual Language Models: A Review

By Tara Safavi, Danai Koutra. 2021

63 citations recorded by the source corpus.

Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which limit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.

Subfields: Artificial Intelligence, Management Science and Operations Research, Large Language Models, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Large Language Models

In Knowledge Graphs & Representation

Yonatan Belinkov

Citations in the corpus

Mass-Editing Memory in a Transformer (2022, background): “Nevertheless, LLMs are promising because they scale well and are unconstrained by a fixed schema (Safavi & Koutra, 2021).”
A Review on Language Models as Knowledge Bases (2022, background): “…Wei et al. (2021) evaluate knowledge-enhanced pretrained LMs by delineating the types of knowledge that can be integrated into existing LMs. Safavi and Koutra (2021) divide relevant work according to the level of supervision provided to the LM by a KB. Similarly, Colon-Hernandez et al.…”