Contextual LSTM (CLSTM) models for Large scale NLP tasks

By Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, Larry Heck. 2016

229 citations recorded by the source corpus.

Documents exhibit sequential structure at multiple levels of abstraction (e.g., sentences, paragraphs, sections). These abstractions constitute a natural hierarchy for representing the context in which to infer the meaning of words and larger fragments of text. In this paper, we present CLSTM (Contextual LSTM), an extension of the recurrent neural network LSTM (Long-Short Term Memory) model, where we incorporate contextual features (e.g., topics) into the model. We evaluate CLSTM on three specific NLP tasks: word prediction, next sentence selection, and sentence topic prediction. Results from experiments run on two corpora, English documents in Wikipedia and a subset of articles from a recent snapshot of English Google News, indicate that using both words and topics as features improves performance of the CLSTM models over baseline LSTM models for these tasks. For example on the next sentence selection task, we get relative accuracy improvements of 21% for the Wikipedia dataset and 18% for the Google News dataset. This clearly demonstrates the significant benefit of using context appropriately in natural language (NL) tasks. This has implications for a wide variety of NL applications like question answering, sentence completion, paraphrase generation, and next utterance prediction in dialog systems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization (2018, background): “The idea of capturing document-level semantic information has been previously explored for recurrent neural networks (Mikolov and Zweig, 2012; Ghosh et al., 2016; Dieng et al., 2017), however, we are not aware of any existing convolutional models.”
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, background): “Differences in nearby vs. long-range context suggest that memory models, which feed explicit context representations to the LSTM (Ghosh et al., 2016; Lau et al., 2017), could beneﬁt from representations that speciﬁcally capture information orthogonal to that modeled by the LSTM.”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, methodology): “Prior performance improvements to dialog models using topic information include appending topic as an additional input [23], or extracting topic information using Latent Dirichlet Allocation [24, 25].”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, methodology): “Contextual RNNs have received a lot of attention (Mikolov and Zweig, 2012; Mikolov et al., 2014; Ji et al., 2015; Lin et al., 2015; Ji et al., 2016; Ghosh et al., 2016).”
Natural language processing: state of the art, current trends and challenges (2017, background): “[47] In order to observe the word arrangement in forward and backward direction, bi-directional LSTM is explored by researchers [59].”
Semi-supervised Variational Autoencoders for Sequence Classification (2016, methodology): “In addition we have also investigated the model which receives the concatenation of word embedding and label vector as in [Ghosh et al., 2016], but it performed a little worse than conditional LSTM by about 2%.”
On-The-Fly Information Retrieval Augmentation for Language Models (2020, background): “Various language models have utilized external knowledge or long contexts (Paperno et al., 2016; Yang and Mitchell, 2017; Peng et al., 2019; Khandelwal et al., 2018; Ghosh et al., 2016; Lau et al., 2017; Grave et al., 2016; Parthasarathi and Pineau, 2018).”
Variational Autoencoder for Semi-Supervised Text Classification (2017, background): “, feeding label y at each time step as in (Wen et al. 2015; Ghosh et al. 2016).”
Inner Attention based Recurrent Neural Networks for Answer Selection (2016, background): “But in attention based models, the question attention information is introduced to in-ﬂuence the answer sentence representation explicitly, in this way we can improve sentence representation for the speciﬁc target (or topic (Ghosh et al., 2016)).”
Focus Attention: Promoting Faithfulness and Diversity in Summarization (2021, methodology): “Recently, document-level topic information has been used for improving neural language models (Mikolov and Zweig, 2012; Ghosh et al., 2016; Dieng et al., 2017; Karmaker Santu et al., 2019), neural response generators (Xing et al., 2017; Dziri et al., 2019), and not surprisingly, neural summarizers…”
Controlling Personality-Based Stylistic Variation with Neural Natural Language Generators (2018, background): “Ghosh et al. (2016) proposed a modiﬁed LSTM cell with an additional gate that incorporates the previous context as input during encoding.”
Cross-topic Argument Mining from Heterogeneous Sources (2018, background): “A more direct approach to integrating an argument’s topic is the contextual LSTM (CLSTM) architecture (Ghosh et al., 2016), where topic information is added as another term to all four gates of an LSTM cell.”
TNT-NLG , System 2 : Data Repetition and Meaning Representation Manipulation to Improve Neural Generation (2018, background): “…domain makes it a good candidate for challenges like E2E, particularly because much of the previous work on NLG has focused on this domain (Howcroft et al., 2013; Polifroni et al., 1992; Whittaker et al., 2002; Stent et al., 2004; Devillers et al., 2004; Gašic et al., 2008; Mairesse et…”
Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record (2018, background): “Many variants of the RNN network have been proposed to address this issue, and long short term memory (LSTM) is one of the most popular models used nowadays in NLP tasks [7], [8], [13]– [16].”
Topic Augmented Neural Response Generation with a Joint Attention Mechanism (2016, background): “J. Gu(Gu et al., 2016) proposed copying mechanism to mimic the repeating phenomenon during human conversation.”