Neural Net Models for Open-Domain Discourse Coherence

By Jiwei Li, Dan Jurafsky. 2016

132 citations recorded by the source corpus.

Discourse coherence is strongly associated with text quality, making it important to natural language generation and understanding. Yet existing models of coherence focus on measuring individual aspects of coherence (lexical overlap, rhetorical structure, entity centering) in narrow domains. In this paper, we describe domain-independent neural models of discourse coherence that are capable of measuring multiple aspects of coherence in existing sentences and can maintain coherence while generating new sentences. We study both discriminative models that learn to distinguish coherent from incoherent discourse, and generative models that produce coherent text, including a novel neural latent-variable Markovian generative model that captures the latent discourse dependencies between sentences in a text. Our work achieves state-of-the-art performance on multiple coherence evaluations, and marks an initial step in generating coherent texts given discourse contexts.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “Table 1: The binary ORDER objective. Discourse relation labels are provided for the reader, but are not available to the model. Looking beyond work on unsupervised pretraining: Li and Hovy (2014) and Li and Jurafsky (2016) use representation learning systems to directly model the problem of sentence”
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents (2019, background): “• (i) the wildly inconsistent outputs and the lack of a consistent personality (Li and Jurafsky, 2016),”
Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems (2019, methodology): “Prior performance improvements to dialog models using topic information include appending topic as an additional input [23], or extracting topic information using Latent Dirichlet Allocation [24, 25].”
Modeling Naive Psychology of Characters in Simple Commonsense Stories (2018, background): “Similarly, there is also a body of research in reasoning about commonsense stories and discourse (Li and Jurafsky, 2017; Mostafazadeh et al., 2016) or detecting emotional stimuli in stories (Gui et al., 2017).”
Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks (2018, methodology): “The sentence ordering task (Li and Jurafsky, 2017) refers to arranging a set of sentences into a coher-”
Sentence Ordering using Recurrent Neural Networks (2016, unknown): “On the EARTHQUAKE data the window approach of Li and Hovy (2014) and Li and Jurafsky (2016) perform strongly.”
Diversifying Dialogue Generation with Non-Conversational Text (2020, methodology): “The typical way of using scores from forward or backward models as in Li and Jurafsky (2017) for the seq2seq model trained on only on the conversational corpus and thus would be assigned very low scores.”
TIARA: A Tool for Annotating Discourse Relations and Sentence Reordering (2020, background): “There are however some studies that explain how to order sentences to generate coherent texts in NLP applications, (Barzilay et al., 2002; Okazaki et al., 2004; Li and Jurafsky, 2017; Xu et al., 2019).”
Coherence Models for Dialogue (2018, background): “Other recent works inspired by the entity grid include coherent paragraph generation [31], and applications to automated essay scoring [32] and neural stories text generation [33].”
Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity (2018, methodology): “• We introduce a stochastic latent variable z conditioned on previous dialogue context to store the global information about the conversation (Bowman et al., 2016; Chung et al., 2015; Li and Jurafsky, 2017; Hu et al., 2017).”