Larger-Context Language Modelling with Recurrent Neural Network

By Tian Wang, Kyunghyun Cho. 2015

88 citations recorded by the source corpus.

In this work, we propose a novel method to incorporate corpus-level discourse information into language modelling. We call this larger-context language model. We introduce a late fusion approach to a recurrent language model based on long short-term memory units (LSTM), which helps the LSTM unit keep intra-sentence dependencies and inter-sentence dependencies separate from each other. Through the evaluation on three corpora (IMDB, BBC, and PennTree Bank), we demon- strate that the proposed model improves perplexity significantly. In the experi- ments, we evaluate the proposed approach while varying the number of context sentences and observe that the proposed late fusion is superior to the usual way of incorporating additional inputs to the LSTM. By analyzing the trained larger- context language model, we discover that content words, including nouns, adjec- tives and verbs, benefit most from an increasing number of context sentences. This analysis suggests that larger-context language model improves the unconditional language model by capturing the theme of a document better and more easily.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “Recurrent Neural Networks based LMs employ the chain rule to model joint probabilities over word sequences: where the context of all previous words is encoded with an LSTM, and the probability over words uses a Softmax (see Figure 1(a)).”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “Wang and Cho (2016) train sentence representations for use as context in language modeling.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “Existing works range from ones where context representations are manually deﬁned (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2015) to others that rely on document-level topics learned from data (Dieng et al., 2016; Wang et al., 2017).”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “…able to keep information from the broader context into memory, and possibly even perform simple forms of reasoning about it (Hermann et al., 2015; Hochreiter and Schmidhuber, 1997; Ji et al., 2015; Mikolov et al., 2015; Sordoni et al., 2015; Sukhbaatar et al., 2015; Wang and Cho, 2015, a.o.).”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “Similarly, topic features have been used with either maximum entropy models (Khudanpur & Wu, 2000) or recurrent networks (Mikolov & Zweig, 2012; Wang & Cho, 2015).”
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, neutral): “In addition, these ﬁndings are consistent with those previously reported for different language models and datasets (Hill et al., 2016; Wang and Cho, 2016).”
Learning Generic Sentence Representations Using Convolutional Neural Networks (2016, neutral): “Similar work to this hierarchical language modeling can be found in Li et al. (2015); Sordoni et al. (2015); Lin et al. (2015); Wang and Cho (2016).”
Natural Language to Code Generation in Interactive Data Science Notebooks (2022, background): “Intuitively, problems located later in a notebook (n is larger) would have more context available, therefore they could be easier to answer (Wang and Cho, 2016).”
Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments (2018, background): “Cross-sentence context has already proven useful for language modeling (Ji et al., 2015; Wang and Cho, 2016) and dialogue systems (Serban et al., 2016).”
Distraction-Based Neural Networks for Modeling Document (2016, methodology): “…can be extended to help model larger spans of text, e.g., documents, is intriguing, and further investigation would still be desirable, although there has been interesting research conducted recently along this line [Li et al., 2015; Hu et al., 2015; Wang and Cho, 2015; Hermann et al., 2015].”
Modeling Coherence for Neural Machine Translation with Dynamic and Topic Caches (2017, methodology): “Jean et al. (2017) propose a novel larger-context neural machine translation model based on the recent works on larger-context language modelling (Wang and Cho, 2016) and employ the method to model the surrounding text in addition to the source sentence.”
Blackbox Meets Blackbox: Representational Similarity & Stability Analysis of Neural Language Models and Brains (2019, unknown): “Interestingly, for UniSentEnc, BERT(L11) and also GoogleLM(L1), increasing the context length, which usually boosts the performance of language encoding models in language understanding tasks (Wang and Cho, 2016), leads to lower similarity with brain representations.”
Topic Compositional Neural Language Model (2017, background): “• LCLM (Wang and Cho, 2016): A context-based language model, which incorporates context information from preceding sentences.”
Topically Driven Neural Language Model (2017, methodology): “Wang and Cho (2016) and Ji et al. (2016) re-lax the sentence independence assumption in language modelling, and use preceeding sentences as additional context.”
Learning to Remember Translation History with a Continuous Cache (2017, background): “…space using a neural network, has a noticeable effect in various deep learning based NLP tasks, such as language modeling (Ji et al., 2015; Wang and Cho, 2016), query suggestion (Sordoni et al., 2015), dialogue modeling (Vinyals and Le, 2015; Serban et al., 2016), and machine translation…”
A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models (2016, background): “Wang and Cho (2015) and Lin et al. (2015) construct bag-of-words representations of previous sentences, which are then used to inform the RNN language model that generates the current sentence.”
Do Long-Range Language Models Actually Use Long-Range Context? (2021, methodology): “Our work is also related to methods that incorporate long context (Wang and Cho, 2016) as well as document-level tasks that inherently require modeling long-range context (Zhang et al.”
Bidirectional Recurrent Neural Network with Attention Mechanism for Punctuation Restoration (2016, methodology): “To fuse together the model state at current input word and the output from the attention mechanism we use a late fusion approach [28] adapted from LSTM to GRU.”
Calibration, Entropy Rates, and Memory in Language Models (2019, background): “…several ways to better capture longterm dependencies, from using manually-defined context representations (Mikolov & Zweig, 2012; Ji et al., 2015; Wang & Cho, 2016) or document-level topics (Wang et al., 2017) to using LSTM recurrent neural networks with careful initialization (Le et al., 2015),…”