A Model of Coherence Based on Distributed Sentence Representation

By Jiwei Li, Eduard H. Hovy. 2014

152 citations recorded by the source corpus.

Coherence is what makes a multi-sentence text meaningful, both logically and syntactically. To solve the challenge of ordering a set of sentences into coherent order, existing approaches focus mostly on defining and using sophisticated features to capture the cross-sentence argumentation logic and syntactic relationships. But both argumentation semantics and crosssentence syntax (such as coreference and tense rules) are very hard to formalize. In this paper, we introduce a neural network model for the coherence task based on distributed sentence representation. The proposed approach learns a syntacticosemantic representation for sentences automatically, using either recurrent or recursive neural networks. The architecture obviated the need for feature engineering, and learns sentence representations, which are to some extent able to capture the ‘rules’ governing coherent sentence structure. The proposed approach outperforms existing baselines and generates the stateof-art performance in standard coherence evaluation tasks 1 .

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, neutral): “Related to our objective is the local coherence model of Li & Hovy (2014) where a binary classiﬁer is trained to identify coherent/incoherent sentence windows.”
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “r uncle, you join Ben. Table 1: The binary ORDER objective. Discourse relation labels are provided for the reader, but are not available to the model. Looking beyond work on unsupervised pretraining: Li and Hovy (2014) and Li and Jurafsky (2016) use representation learning systems to directly model ”
Learning to Extract Coherent Summary via Deep Reinforcement Learning (2018, methodology): “(Li and Hovy 2014) use the recurrent and recursive neural network to obtain the distributed representation of sentences and then use a pairwise ranking method to train the coherence model.”
A Hierarchical Neural Autoencoder for Paragraphs and Documents (2015, background): “Although neural representations for sentences can capture aspects of coherent sentence structure (Ji and Eisenstein, 2014; Li et al., 2014; Li and Hovy, 2014), it’s not clear how they could help in generating more broadly coherent text.”
Document Context Language Models (2015, methodology): “Therefore, we have the same computation procedure on perplexity for the models with and without contextual information.”
Sentence Ordering using Recurrent Neural Networks (2016, unknown): “On the EARTHQUAKE data the window approach of Li and Hovy (2014) and Li and Jurafsky (2016) perform strongly.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “There are also extensions to multiple sentences [163, 199, 200].”
When Are Tree Structures Necessary for Deep Learning of Representations? (2015, methodology): “A discourse relation classiﬁer takes as input a vector embedding representation of each of the two EDUs, and computes the most likely relation (Li et al., 2014).”
Neural Net Models for Open-Domain Discourse Coherence (2016, methodology): “The algorithm we present sig-niﬁcantly outperforms the classiﬁer of Li and Hovy (2014).”
A Neural Local Coherence Model (2017, methodology): “Distributed Sentence Model: Li and Hovy (2014) proposed this neural model for measuring 6https://bitbucket.org/melsner/browncoherence 7https://github.com/karins/CoherenceFramework text coherence.”
Discourse-Aware Neural Rewards for Coherent Text Generation (2018, background): “…from previous research that learns the overall ordering structure of a document as an approximation of the discourse structure (Barzilay and Lapata, 2005, 2008; Barzilay and Lee, 2004; Li and Hovy, 2014), and propose two neural teachers that can learn to score an ordered sequence of sentences.”
Hierarchical Recurrent Neural Network for Document Modeling (2015, background): “Liu et al. (2014) propose a recursive recurrent neural network (R2NN) for end-to-end decoding to help improve translation quality.”
Sentiment Embeddings with Applications to Sentiment Analysis (2016, methodology): “For parameter update, we use AdaGrad [58], which is widely applied in deep learning literature [59], [60], [61].”
Pre-Trained Language Models for Text Generation: A Survey (2022, background): “In linguistics [125], language coherence is what makes a multi-sentence text meaningful, both logically and syntactically.”