Document Context Language Models

By Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, Jacob Eisenstein. 2015

84 citations recorded by the source corpus.

Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “Much work has been done on small and large scale RNN-based LMs (Mikolov et al., 2010; Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “In addition, Ji et al. (2016) treat discourse relations between sentences as latent variables and show that this yields improvements in language modeling in an extension of the document-context model of Ji et al. (2015).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “ually defined (Mikolov and Zweig, 2012; Ji et al., 2015; Wang and Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al.”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “…able to keep information from the broader context into memory, and possibly even perform simple forms of reasoning about it (Hermann et al., 2015; Hochreiter and Schmidhuber, 1997; Ji et al., 2015; Mikolov et al., 2015; Sordoni et al., 2015; Sukhbaatar et al., 2015; Wang and Cho, 2015, a.o.).”
Controlling Politeness in Neural Machine Translation via Side Constraints (2016, background): “Previous research on neural language models has proposed including various types of extra information, such as topic, genre or document context (Mikolov and Zweig, 2012; Aransa et al., 2015; Ji et al., 2015; Wang and Cho, 2015).”
Reference-Aware Language Models (2016, background): “Finally, we address references within a document (Mikolov et al., 2010; Ji et al., 2015; Wang and Cho, 2015), as the generation of words will ofter refer to previously generated words.”
Larger-Context Language Modelling with Recurrent Neural Network (2015, neutral): “∗Recently, (Ji et al., 2015) independently proposed a similar approach.”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, background): “Contextual RNN s have received a lot of attention (Mikolov and Zweig, 2012; Mikolov et al., 2014; Ji et al., 2015; Lin et al., 2015; Ji et al., 2016; Ghosh et al., 2016).”
Dynamic Entity Representations in Neural Language Models (2017, background): “Previous work focuses on contextual information from previous sentences (Ji et al., 2016a) or discourse relations be-tween adjacent sentences (Ji et al., 2016b), showing improvements to language modeling and re-lated tasks like coherence evaluation and discourse relation prediction.”
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories (2018, background): “account of left context from both the current sentence and the previous sentence [19].”
Does Neural Machine Translation Benefit from Larger Context? (2017, methodology): “For instance, discourse coherence has been used as one of such metrics in analyzing larger-context language modelling in recent years (Ji et al., 2015, 2016).”
Do Long-Range Language Models Actually Use Long-Range Context? (2021, background): “Many research efforts over the years have attempted to address this issue: for example, Rosenfeld (1996) incorporated statistics from distant tokens to improve n -gram models, while Ji et al. (2015) added document-level context to neural LMs.”
Calibration, Entropy Rates, and Memory in Language Models (2019, background): “Recent approaches to improving language modeling have focused on several ways to better capture longterm dependencies, from using manually-defined context representations (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2016) or document-level topics (Wang et al.”
A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models (2016, methodology): “Ji et al. (2015) propose an alternative form of the document context language model, in which the contextual information c t impacts the hidden state h t +1 , rather than going directly to the outputs y t +1 .”
Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments (2018, background): “Cross-sentence context has already proven useful for language modeling (Ji et al., 2015; Wang and Cho, 2016) and dialogue systems (Serban et al.”
Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “Existing works range from ones where context representations are manually deﬁned (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
Learning to Remember Translation History with a Continuous Cache (2017, background): “Exploitation of Cross-Sentence Context Crosssentence context, which is generally encoded into a continuous space using a neural network, has a noticeable effect in various deep learning based NLP tasks, such as language modeling (Ji et al., 2015; Wang and Cho, 2016), query suggestion (Sordoni et al.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, unknown): “producing character-level outputs [55], [64]–[66].”