Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

By IV RobertL.Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, Sameer Singh. 2019

203 citations recorded by the source corpus.

Modeling human language requires the ability to not only generate fluent text but also encode factual knowledge. However, traditional language models are only capable of remembering facts seen at training time, and often have difficulty recalling them. To address this, we introduce the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context. These mechanisms enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. We also introduce the Linked WikiText-2 dataset, a corpus of annotated text aligned to the Wikidata knowledge graph whose contents (roughly) match the popular WikiText-2 benchmark. In experiments, we demonstrate that the KGLM achieves significantly better performance than a strong baseline language model. We additionally compare different language model’s ability to complete sentences requiring factual knowledge, showing that the KGLM outperforms even very large language models in generating facts.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “Knowledge-augmented language models Our work shares inspiration with other approaches seeking to inject knowledge into language models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wa”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “†Equal contribution. matical correctness of text decoded from modern LMs. Additionally, recent works (Petroni et al., 2019; Logan et al., 2019; Broscheit, 2019; Roberts et al., 2020) demonstrate that beyond general linguistic capabilities, language models can also pick up factual knowledge present…”
Knowledge Enhanced Contextual Word Representations (2019, background): “Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, background): “…neural LMs often repeat themselves (Holtzman et al., 2020), suffer from overconﬁ-dence (Braverman et al., 2020; Jiang et al., 2020a), suffer from recency bias (Khandelwal et al., 2018; Ravfogel et al., 2019), and prefer generic responses instead of rare text (Li et al., 2016; Logan et al., 2019).”
Unsupervised Commonsense Question Answering with Self-Talk (2020, background): “Speciﬁcally, Kassner and Sch¨utze (2019) showed that negated facts are also considered likely by the LM, while Logan et al. (2019) pointed out that LMs may over-generalize and produce incorrect facts such as “Barack Obama’s wife is Hillary”.”
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, methodology): “Evaluation Metrics Previous work (Logan et al., 2019; Petroni et al., 2019) either relies on human evaluation or only considers single-token entities for fact completion.”
Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge (2020, methodology): “Unlike prior models, EaE learns entity representations end-to-end, rather than using representations from a separately-trained KB embedding system (Logan et al., 2019).”
Knowledge-Aware Language Model Pretraining (2020, background): “The language models still (de-)generate dull, factually-incorrect, or dream-like text when used in natural language generation [13, 14, 15, 16].”
Visually-Augmented Language Modeling (2022, background): “Since current unimodal PLMs lack visual knowledge grounding, they inevitably suffer from the hallucination problem, which refers to the inconsistent or false statements generated by PLMs with respect to the world knowledge (Logan et al., 2019).”
AggGen: Ordering and Aggregating while Generating (2021, methodology): “, 2020) or decoding of explicit graph references (Logan et al., 2019) are applied for higher accuracy.”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “KGLM [113] selects the facts from a knowledge graph using the current context to generate factual sentences.”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “[25] Robert L Logan IV, Nelson F Liu, Matthew E Peters, Matt Gardner, and Sameer Singh.”