Dynamic Entity Representations in Neural Language Models

By Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, Noah A. Smith. 2017

92 citations recorded by the source corpus.

Understanding a long document requires tracking how entities are introduced and evolve over time. We present a new type of language model, EntityNLM, that can explicitly model entities, dynamically update their representations, and contextually generate their mentions. Our model is generative and flexible; it can model an arbitrary number of entities in context while generating each entity mention at an arbitrary length. In addition, it can be used for several different tasks such as language modeling, coreference resolution, and entity prediction. Experimental results with all these tasks demonstrate that our model consistently outperforms strong baselines and prior work.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Knowledge Enhanced Contextual Word Representations (2019, background): “Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…neural architectures on text with entity coreference information yields more entity-aware LLMs, both in recurrent (Henaff et al., 2017; Ji et al., 2017; Yang et al., 2017; Liu et al., 2019) and Transformer-based models (Févry et al., 2020; De Cao et al., 2020; Rosset et al., 2020;…”
Linguistic Knowledge and Transferability of Contextual Representations (2019, neutral): “To alleviate this weakness, future work could augment pretrained contextual-izers with explicit entity representations (Ji et al., 2017; Yang et al., 2017; Bosselut et al., 2017).”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, methodology): “Following Ji et al. (2017), we compute q ( E | x ) using a discriminative version of our model that predicts annotations for the current token instead of for the next token.”
Modeling Naive Psychology of Characters in Simple Commonsense Stories (2018, background): “Entity modeling in these works, however, was limited to tracking entity reference (Kiddon et al., 2016; Yang et al., 2016; Ji et al., 2017), recognizing entity state similarity (Henaff et al.”
Context-Aware Neural Machine Translation Learns Anaphora Resolution (2018, background): “One option would be to integrate ‘global’ features summarizing properties of groups of mentions predicted as linked in a document (Wiseman et al., 2016), or to use latent relations to trace entities across documents (Ji et al., 2017).”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, background): “The main work we are aware of is (Dhingra et al. 2018), which uses coreference clusters to improve reading comprehension and the works on language modeling by Ji et al. (2017) and Yang et al. (2016).”
Published as a conference paper at ICLR 2018 S IMULATING A CTION D YNAMICS WITH N EURAL P ROCESS N ETWORKS (2018, methodology): “Our model contributes to a recent line of research that aims to model aspects of world state changes, such as language models and machine readers with explicit entity representations (Henaff et al., 2016; Yang et al., 2016; Ji et al., 2017), as well as other more general purpose memory network”
Question Answering by Reasoning Across Documents with Graph Convolutional Networks (2018, background): “Dynamic representations of entities, modeled in a generative framework, have been studied in the context of language modeling (Ji et al., 2017), where they were shown effective in capturing long-distance dependencies in a document.”
End-to-End Neural Entity Linking (2018, methodology): “We here advocate for models that address the end-to-end EL task, informally arguing that humans understand and generate text in a similar joint manner, discussing about entities which are gradually introduced, referenced under multiple names and evolving during time (Ji et al., 2017).”
Latent Relation Language Models (2019, background): “A variety of entity-aware LMs exist, conditioning on a variety of information sources such as expert coreference annotations (Ji et al., 2017; Clark et al., 2018; Yang et al., 2017), entity annotations (Logan et al., 2019), deﬁnitions (Bah-danau et al., 2017), or keywords (Kiddon et al., 2016;…”
Learning to Select, Track, and Generate for Data-to-Text (2019, background): “Ji et al. (2017) proposed an entity representation for the language model.”
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020, background): “For this task, prior studies have attempted to build speciﬁc sentence representations by modeling story entities and events to simplify the dependencies between sentences (Ji et al., 2017; Clark et al., 2018).”
Dynamic Integration of Background Knowledge in Neural NLU Systems (2017, background): “Tracking and updating concepts, entities or sentences with dynamic memories is a very active research direction (Kumar et al., 2016; Henaff et al., 2017; Ji et al., 2017; Kobayashi et al., 2017).”
Story Ending Selection by Finding Hints From Pairwise Candidate Endings (2019, methodology): “Rulebased methods [37], [38] and neural models with Sequence-toSequence framework [39], [40] are widely used.”
Data-to-text Generation with Entity Modeling (2019, methodology): “Ji et al. (2017) make use of dynamic entity representations for language modeling.”
The Referential Reader: A Recurrent Entity Network for Anaphora Resolution (2019, background): “Ji et al. (2017) propose a generative entity-aware language model that incorporates coreference as a discrete latent variable.”
Neural Models for Reasoning over Multiple Mentions Using Coreference (2018, methodology): “Ji et al. (2017) presented a generative model for jointly predicting the next word in the text and its gold-standard coreference annotation.”
Relational Memory-Augmented Language Models (2022, methodology): “There are language models incorporating entity information, such as entity coreference annotations (Ji et al., 2017; Clark et al., 2018), surface forms of entities (Kiddon et al.”