Reference-Aware Language Models

By Zichao Yang, Phil Blunsom, Chris Dyer, Wang Ling. 2016

81 citations recorded by the source corpus.

We propose a general class of language models that treat reference as discrete stochastic latent variables. This decision allows for the creation of entity mentions by accessing external databases of referents (required by, e.g., dialogue generation) or past internal state (required to explicitly model coreferentiality). Beyond simple copying, our coreference model can additionally refer to a referent using varied mention forms (e.g., a reference to “Jane” can be realized as “she”), a characteristic feature of reference in natural languages. Experiments on three representative applications show our model variants outperform models based on deterministic attention and standard language modeling baselines.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Entities as Experts: Sparse Memory Access with Entity Supervision (2020, background): “…language models Our work shares inspiration with other approaches seeking to inject knowledge into language models (Ahn et al., 2016; Yang et al., 2016; Logan et al., 2019; Zhang et al., 2019; Levine et al., 2019; Xiong et al., 2019a; Peters et al., 2019; Poerner et al., 2019; Wang…”
How Can We Know What Language Models Know? (2019, background): “Orthogonally, some previous works integrate external knowledge bases so that the language generation process is explicitly conditioned on symbolic knowledge (Ahn et al., 2016; Yang et al., 2017; Logan et al., 2019; Hayashi et al., 2020).”
Knowledge Enhanced Contextual Word Representations (2019, background): “Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Hinting at this, prior work has shown that training entity-centric neural architectures on text with entity coreference information yields more entity-aware LLMs, both in recurrent (Henaff et al., 2017; Ji et al., 2017; Yang et al., 2017; Liu et al., 2019) and Transformerbased models (Févry et al.”
Challenges in Data-to-Document Generation (2017, methodology): “Copying There has been a surge of recent work involving augmenting encoder-decoder models to copy words directly from the source material on which they condition (Gu et al., 2016; G¨ulc¸ehre et al., 2016; Merity et al., 2016; Jia and Liang, 2016; Yang et al., 2016).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, neutral): “To alleviate this weakness, future work could augment pretrained contextual-izers with explicit entity representations (Ji et al., 2017; Yang et al., 2017; Bosselut et al., 2017).”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, methodology): “Our goals are much more aligned to the data-to-text task (Ahn et al., 2016; Lebret et al., 2016; Wiseman et al., 2017; Yang et al., 2017; Gardent et al., 2017; Ferreira et al., 2018), where a small table-sized KB is provided to generate a short piece of text; we are interested in language models…”
Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge (2018, background): “Yang et al. (2017) extended a seq2seq model with attention to external facts for dialogue and recipe generation and a co-reference resolution-aware language model.”
Modeling Naive Psychology of Characters in Simple Commonsense Stories (2018, background): “Entity modeling in these works, however, was limited to tracking entity reference (Kid-don et al., 2016; Yang et al., 2016; Ji et al., 2017), recognizing entity state similarity (Henaff et al., 2017) or predicting simple attributes from entity states (Bosselut et al., 2018).”
Dynamic Entity Representations in Neural Language Models (2017, background): “eciﬁcally to only one entity. Entity-related models. Two recent approaches to modeling entities in text are closely related to our model. The ﬁrst is the “reference-aware” language models proposed by Yang et al. (2016), where the referred entities are from either a predeﬁned item list, an external d”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, background): “The main work we are aware of is (Dhingra et al. 2018), which uses coreference clusters to improve reading comprehension and the works on language modeling by Ji et al. (2017) and Yang et al. (2016).”
Data-to-Text Generation with Content Selection and Planning (2018, methodology): “Following previous work (Yang et al. 2017; Wiseman et al. 2017), we embed features into vectors, and then use a multilayer perceptron to obtain a vector representation r j for each record: r j = ReLU( W r [ r j, 1 ; r j, 2 ; r j, 3 ; r j, 4 ] + b r ) where [; ] indicates vector concatenation, W r ∈…”
Neural Models for Reasoning over Multiple Mentions Using Coreference (2018, background): “Yang et al. (2017) model references in text as explicit latent variables, but limit their work to text generation.”
Discourse-Aware Neural Rewards for Coherent Text Generation (2018, methodology): “…and Schmid-huber, 1997; Cho et al., 2014) are trained using cross-entropy loss (Bahdanau et al., 2015; Vinyals et al., 2015; Xu et al., 2015; Rush et al., 2015), often augmented with additional terms for topic coverage or task-speciﬁc supervision (Kiddon et al., 2016; Yang et al., 2017).”
SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning (2017, methodology): “The decoder phase computes the distribution of the next token using a pointer network (Vinyals et al., 2015a; Yang et al., 2016) with the column attention mechanism.”
Linguistic Knowledge as Memory for Recurrent Neural Networks (2017, unknown): “Recently, there has been interest in incorporating symbolic knowledge, such as that from a Knowledge Base or coreference information, within RNN-based language models (Yang et al., 2016; Ahn et al., 2016).”
Relational Memory-Augmented Language Models (2022, background): “, 2018), surface forms of entities (Kiddon et al., 2016; Yang et al., 2017; Cao et al., 2021), entity types (Parvez et al.”
RecipeNLG: A Cooking Recipes Dataset for Semi-Structured Text Generation (2020, methodology): “(Yang et al., 2017) used a dataset with 31K recipes to propose reference-aware language models to generate instructions based on the ingredients provided.”