Latent Relation Language Models

By Hiroaki Hayashi, Zecong Hu, Chenyan Xiong, Graham Neubig. 2019

43 citations recorded by the source corpus.

In this paper, we propose Latent Relation Language Models (LRLMs), a class of language models that parameterizes the joint distribution over the words in a document and the entities that occur therein via knowledge graph relations. This model has a number of attractive properties: it not only improves language modeling performance, but is also able to annotate the posterior probability of entity spans for a given text through relations. Experiments demonstrate empirical improvements over both word-based language models and a previous approach that incorporates knowledge graph information. Qualitative analysis further demonstrates the proposed model's ability to learn to predict appropriate relations in context. 1

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Pre-trained models for natural language processing: A survey (2020, background): “[56] extended language model to knowledge graph language model (KGLM) and latent relation language model (LRLM) respectively, both of which allow prediction conditioned on knowledge graph.”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “Howard and Ruder (2018) propose a universal language model (ULMFiT) based on AWD-LSTM (Merity et al., 2018).”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, background): “That is, they can produce logically or factually inaccurate, or contradicting statements (Welleck et al., 2018; Zhang et al., 2018; Hayashi et al., 2019; Petroni et al., 2019).”
HittER: Hierarchical Transformers for Knowledge Graph Embeddings (2020, background): “Because of their broad applications in various intelligent systems including natural language understanding (Logan et al., 2019; Zhang et al., 2019b; Hayashi et al., 2020) and reasoning (Riedel et al.”
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, background): “Lots of work has been done on integrating knowledge from the general domain into models (Zhang et al., 2019; Peters et al., 2019; Hayashi et al., 2019).”
Relational Memory-Augmented Language Models (2022, methodology): “Our model is closely related to previous work on grounding autoregressive language models with knowledge graphs (Ahn et al., 2016; Logan et al., 2019; Hayashi et al., 2020; Wang et al., 2021a).”
Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs (2019, background): “The structured knowledge graphs include rich semantics represented via entities and relations (Hayashi et al., 2019).”
Knowledge-Aware Language Model Pretraining (2020, methodology): “To address this, previous research mainly grounds the language model to some forms of explicit semantic structures, for example, additional entity-oriented transformer layers [16, 18], or explicitly stored knowledge graph triples [19, 40, 41].”