A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models

By Yangfeng Ji, Gholamreza Haffari, Jacob Eisenstein. 2016

123 citations recorded by the source corpus.

This paper presents a novel latent variable recurrent neural network architecture for jointly modeling sequences of words and (possibly latent) discourse relations between adjacent sentences. A recurrent neural network generates individual words, thus reaping the benefits of discriminatively-trained vector representations. The discourse relations are represented with a latent variable, which can be predicted or marginalized, depending on the task. The resulting model can therefore employ a training objective that includes not only discourse relation classification, but also word prediction. As a result, it outperforms state-of-the-art alternatives for two tasks: implicit discourse relation classification in the Penn Discourse Treebank, and dialog act classification in the Switchboard corpus. Furthermore, by marginalizing over latent discourse relations at test time, we obtain a discourse informed language model, which improves over a strong LSTM baseline.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “In addition, Ji et al. (2016) treat discourse relations between sentences as latent variables and show that this yields improvements in language modeling in an extension of the document-context model of Ji et al. (2015).”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, background): “These topics provide context to the RNN. Contextual RNNs have received a lot of attention (Mikolov and Zweig, 2012; Mikolov et al., 2014; Ji et al., 2015; Lin et al., 2015; Ji et al., 2016; Ghosh et al., 2016).”
Dynamic Entity Representations in Neural Language Models (2017, background): “Previous work focuses on contextual information from previous sentences (Ji et al., 2016a) or discourse relations be-tween adjacent sentences (Ji et al., 2016b), showing improvements to language modeling and re-lated tasks like coherence evaluation and discourse relation prediction.”
Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts (2018, methodology): “A dialogue instance in the dataset built from the AMI meeting corpus. rent neural network (RNN) [33, 34, 35, 36].”
Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains (2019, methodology): “…models, including feed-forward network (Rutherford et al., 2017), convolutional neural networks (Zhang et al., 2015), recurrent neural network (Ji et al., 2016; Bai and Zhao, 2018), character-based (Qin et al., 2016) or formulating relation classiﬁcation as an adversarial task (Qin et al.,…”
Recent advances in document summarization (2017, background): “Explicitly designing latent variable structures to capture discourse relations between sentences [84] may also help the document encoding process.”
Does Neural Machine Translation Benefit from Larger Context? (2017, methodology): “For instance, discourse coherence has been used as one of such metrics in analyzing larger-context language modelling in recent years (Ji et al., 2015, 2016).”
Multiresolution Recurrent Neural Networks: An Application to Dialogue Response Generation (2016, methodology): “Closely related to our work is the model proposed by Ji et al.[12], which jointly models natural language text and high-level discourse phenomena.”
Restructuring Conversations using Discourse Relations for Zero-shot Abstractive Dialogue Summarization. (2019, background): “(CRF) (Okazaki, 2007) in this paper, which has been recognized as a good baseline for discourse labelling (Kumar et al., 2017; Ji et al., 2016; Lee and Dernoncourt, 2016).”