A Neural Knowledge Language Model

By Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, Yoshua Bengio. 2016

124 citations recorded by the source corpus.

Current language models have significant limitations in their ability to encode and decode knowledge. This is mainly because they acquire knowledge based on statistical co-occurrences, even if most of the knowledge words are rarely observed named entities. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model. At each time step, the model predicts a fact on which the observed word is to be based. Then, a word is either generated from the vocabulary or copied from the knowledge graph. We train and test the model on a new dataset, WikiFacts. In experiments, we show that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words. In addition, we demonstrate that the sampled descriptions include named entities which were used to be the unknown words in RNN language models.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Improving language models by retrieving from trillions of tokens (2021, methodology): “The retrieval database may also be structured — for example, Ahn et al. (2016) use a symbolic knowledge graph to improve an RNN language model.”
Pointer Sentinel Mixture Models (2016, background): “…to increase capacity and minimize the impact of vanishing gradients, some language and translation mod-Pointer els have also added a soft attention or memory component (Bahdanau et al., 2015; Sukhbaatar et al., 2015; Cheng et al., 2016; Kumar et al., 2016; Xiong et al., 2016; Ahn et al., 2016).”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “Knowledge-augmented language models Our work shares inspiration with other approaches seeking to inject knowledge into language models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wa”
How Can We Know What Language Models Know? (2019, background): “Orthogonally, some previous works integrate external knowledge bases so that the language generation process is explicitly conditioned on symbolic knowledge (Ahn et al., 2016; Yang et al., 2017; Logan et al., 2019; Hayashi et al., 2020).”
Knowledge Enhanced Contextual Word Representations (2019, background): “Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
Reference-Aware Language Models (2016, methodology): “Our models are similar to models proposed in (Ahn et al., 2016; Merity et al., 2016), where the generation of each word can be conditioned on a particular entry in knowledge lists and previous words.”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, methodology): “(ii) The neural knowledge language model (NKLM) (Ahn et al., 2016) which established the idea of leveraging knowledge graphs in neural language models.”
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, methodology): “As standard language encoders usually do not explicitly model knowledge, recent studies (Ahn et al., 2016; Yang & Mitchell, 2017; Logan et al., 2019; Liu et al., 2019a) have explored methods to incorporate external knowledge into NLP models.”
Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge (2018, background): “Ahn et al. (2016) proposed a language model that copies fact attributes from a topic knowledge memory.”
Latent Relation Language Models (2019, methodology): “NKLM generates more invalid mentions than LRLM, most of which are false positives and repetitions of the same entity.”
Just Add Functions: A Neural-Symbolic Language Model (2019, background): “One of the more prominent examples is the Neural Knowledge Language Model (NKLM), which composes a knowledge graph with a NNLM (Ahn et al. 2017).”
Representation Learning for Visual-Relational Knowledge Graphs (2017, background): “In recent years KGs have been playing an increasingly crucial role in fields such as question answering [6, 9], language modeling [1], and text generation [40].”
Factuality Enhanced Language Models for Open-Ended Text Generation (2022, background): “[24] Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio.”
Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs (2017, background): “, 2017], language modeling [Ahn et al., 2016], and text generation [Serban et al.”
Relational Memory-Augmented Language Models (2022, methodology): “Our model is closely related to previous work on grounding autoregressive language models with knowledge graphs (Ahn et al., 2016; Logan et al., 2019; Hayashi et al., 2020; Wang et al., 2021a).”
Improving Question Answering by Commonsense-Based Pre-Training (2018, unknown): “A similar idea has also been applied to conversation generation (Zhou et al. 2018), answering complex questions (Khot, Sabharwal, and Clark 2017), task-oriented dialog system (Madotto, Wu, and Fung 2018), text entailment (Chen et al. 2018b), language modeling (Ahn et al. 2016), etc.”
Leveraging Knowledge Bases in LSTMs for Improving Machine Reading (2017, background): “More recently, Ahn et al. (2016) proposed a neural knowledge language model that leverages knowledge bases in RNN language models, which allows for better representations of words for language modeling.”
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, background): “Knowledge-aware LM Text, knowledge (sub)graph Target entity, text Neural Knowledge LM (Ahn et al., 2016), Reference-aware LM (Yang et al.”
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference (2018, methodology): “Ahn et al. (2016) integrate Freebase facts into a language model using a copying mechanism over fact attributes.”